Did My Job Run — Blog

Why silent cron failures are the expensive kind

When a website goes down, you find out fast. Customers complain, someone posts in Slack, an uptime checker fires. The failure is loud, and loud failures get fixed.

Scheduled jobs are the opposite. When a nightly backup stops running, nothing happens. No error page, no angry email, no alert. The job simply doesn’t run, and the absence of an event is not itself an event. You find out weeks later, at the worst possible moment — when you reach for the backup and it isn’t there.

The failure modes that don’t page anyone

A job can stop running for reasons that never produce a stack trace:

  • The server was rebuilt and the crontab didn’t come with it.
  • A dependency upgrade broke the script’s import, and cron mailed the error to a local mailbox nobody reads.
  • The container that ran the job got rescheduled and the schedule was lost.
  • Someone rotated a credential the job needed.
  • The job is still running, but it now takes nine hours instead of twenty minutes, so it silently overlaps with the next run.

Every one of these is invisible to error monitoring, because there is no error. The code didn’t fail. The code didn’t run.

Monitoring for absence, not failure

The fix is to invert the question. Instead of asking “did this job report an error?”, ask “did this job report at all?”

That’s a dead man’s switch. Your job pings a URL when it finishes. If the ping doesn’t arrive inside the window you expect, something is wrong — and now the silence itself is the alert.

Adding it to an existing cron job is one line:

0 3 * * * /usr/local/bin/backup.sh && curl -fsS https://api.didmyjobrun.com/ping/YOUR-UUID

The && matters. The ping only fires if backup.sh exits 0, so a job that runs but fails is treated the same as a job that never ran at all.

What “good” looks like

A scheduled job you can trust has three properties:

  1. It tells you it finished. Success is reported, not assumed.
  2. Someone is told when it doesn’t. The alert reaches a human, on a channel they actually read.
  3. The expected window is written down. “Every night around 3am” is a feeling. “Every 24 hours, with a 1 hour grace period” is a check.

Most teams have the first one covered by accident, through logs nobody reads. The second and third are what turn a pile of cron jobs into something you can stop worrying about.

That’s the whole idea behind Did My Job Run: you keep your jobs exactly as they are, add one curl, and we handle the part where somebody notices.