A dead engine does not stop the job
Two runs on one time axis. Without a watchdog, the engine stopped at start, and the rollout loop then logged 200,533 HTTP 503 errors over 63 minutes while the GPU did nothing, until we stopped the container. With watchdog.sh, a test container that logged the same error without a break stopped after 183 seconds, when six checks in a row found more than 100 errors. A healthy 16-answer run logged bursts of this error of up to 60 seconds, two of them 45 seconds apart. A replay of that run stopped it in more than half of the cases with six checks and in none with ten, so we run the watchdog with ten.
MINUTES AFTER THE ENGINE STOPPED
0
15
30
45
60
no watchdog
the job keeps going
200,533 lines of HTTP 503 from /generate
GPU idle, no failure reported
we stopped it at 63 min
watchdog.sh test
needs 6 bad checks
the watchdog stops the container
exit 1 after 183 s
more than 100 errors in six checks
Start every unattended run with a watchdog. The job alone never fails.
Healthy bursts came 45 s apart, so six checks are too few. Run it with ten.