How to monitor a Kubernetes CronJob without re-running it

The usual way to monitor a Kubernetes CronJob is a heartbeat: the job pings a URL when it finishes, and if that ping doesn’t arrive on time, something is wrong. Silence becomes the alert, which is the only thing that catches a job that never ran at all.

On a plain cron server you add it like this and you’re done:

0 3 * * * /usr/local/bin/backup.sh && curl -fsS https://api.didmyjobrun.com/ping/YOUR-UUID

Move that same line into a Kubernetes CronJob and you’ve introduced a bug.

Why Kubernetes retries a CronJob when cron would not

Cron runs your command and forgets about it. If the command exits non-zero, cron does nothing — maybe it mails a local mailbox nobody reads. That’s the end of it.

Kubernetes doesn’t forget. A Job has a backoffLimit, which defaults to 6. When the pod exits non-zero, the Job controller creates a new pod and runs your container again. And again. That’s usually what you want — a transient failure retries itself while you sleep.

Now look at the exit code of that command.

./backup.sh && curl -fsS "$PING_URL"

With &&, the exit code of the whole thing is curl’s exit code. And -f makes curl exit non-zero on any HTTP error — a 500, a 502 from a proxy, a timeout, a DNS hiccup.

So: your backup succeeds. The monitoring endpoint has a bad thirty seconds. Curl exits 22. The container exits 22. Kubernetes sees a failed Job and runs your backup again. Up to six more times.

Your monitoring just re-ran the thing it was supposed to be watching.

For a backup that’s wasted I/O and a confusing pager. For a job that sends invoices, posts to a payment API, or writes to a ledger, it’s worse than no monitoring at all — you’ve added a way for a network blip on an unrelated service to duplicate real work.

The fix: let the heartbeat ping fail safely

Let the ping fail without taking the job down with it:

command:
  - /bin/sh
  - -c
  - |
    set -e
    ./backup.sh
    curl -fsS --retry 3 --max-time 10 "$PING_URL" \
      || echo "heartbeat failed" >&2

Three things are doing work here:

set -e means a failing backup.sh stops the script before the ping. The monitor stays silent, which is exactly right — a job that ran and failed should look identical to a job that never ran.

|| echo swallows curl’s exit code. The container exits 0, Kubernetes marks the Job complete, nothing retries. You still find out, because the ping never arrived and the grace period will lapse. You just don’t corrupt anything on the way to finding out.

--retry 3 --max-time 10 rides out a blip without hanging the pod if the endpoint is genuinely down.

The asymmetry is the point. A missing ping should page you. A missing ping should never change what your job did.

Where else monitoring changes what it measures

Once you’ve seen it, you start noticing it in other places where monitoring shares a fate with the thing it monitors:

• A sidecar that scrapes metrics and gets OOMKilled, taking the pod with it.
• An init container that checks a dependency and blocks the real work when the check itself is broken.
• A readiness probe hitting a /health endpoint that calls a third-party API — their outage becomes your rolling restart.

The rule generalises: instrumentation should be able to fail without changing the behaviour of the thing it observes. Anywhere your monitoring can vote on whether the real work happened, you’ve built a way for the monitoring to cause an incident instead of reporting one.

That’s the part Did My Job Run is built around: you keep your CronJobs exactly as they are, add one line that can’t hurt them, and we watch for the silence.