Did My Job Run — Blog

How to monitor a systemd timer (list-timers lies to you)

Here’s the check almost everyone runs to see whether a systemd timer is working:

systemctl list-timers
NEXT                        LEFT     LAST                        PASSED  UNIT           ACTIVATES
Tue 2026-07-21 03:00:00 UTC 12h left Mon 2026-07-20 03:00:00 UTC 9h ago  backup.timer   backup.service

Next run scheduled, last run this morning, right on time. Looks healthy.

That output is compatible with backup.service having failed every single night for the past month.

Two units, and you’re watching the wrong one

A systemd timer is always a pair. backup.timer decides when. backup.service does the work. They’re separate units with separate state.

list-timers reports on the timer. The LAST column means “the last time this timer fired its service” — not “the last time the job worked.” The timer’s job is to pull the trigger. It did that. It has no opinion on what happened next.

So the service can start, exit 1 immediately, and land in failed state, and the timer will still show a healthy LAST and a cheerful NEXT. It’ll do it again tomorrow. The output above never changes.

The unit that actually knows is the service:

systemctl status backup.service
systemctl is-failed backup.service

Or, for everything at once — the command worth knowing:

systemctl --failed

Run that on a box you haven’t looked at in a while. It’s often educational.

Why nobody notices

Because failed units are perfectly quiet. A failed unit sits there in red, forever, and systemd’s response is to write a line to the journal and move on. Nothing emails you. Nothing pages you. The next timer cycle fires regardless, so there’s no cascading breakage to trip over.

You have to go looking, and going looking requires already suspecting.

Wire up OnFailure

systemd has a built-in hook for this and it’s underused. In the service unit, not the timer:

[Unit]
Description=Nightly backup
OnFailure=alert@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

When backup.service fails, systemd starts alert@backup.service.service. %n passes the failed unit’s name through, so one template unit covers every job on the box:

# /etc/systemd/system/alert@.service
[Unit]
Description=Alert for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/send-alert.sh %i

That closes the loop on failure. It does nothing for the case that actually hurts.

OnFailure can’t fire if nothing ran

OnFailure needs a failure to react to. It’s silent when:

  • Someone ran systemctl disable --now backup.timer during an incident.
  • The unit file was removed by a config management run.
  • The box was off, and Persistent=true wasn’t set, so the missed run was simply skipped.
  • The whole machine is gone.

In every one of those, nothing failed. Nothing ran. systemctl --failed is clean, OnFailure has nothing to hook, and the job is dead.

For that you need the inverse check — something outside the box expecting to hear from it. ExecStartPost is the right place, because for Type=oneshot it only runs if ExecStart succeeded:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
ExecStartPost=/usr/bin/curl -fsS --retry 3 --max-time 10 https://api.didmyjobrun.com/ping/YOUR-UUID

Backup fails, no ping. Timer disabled, no ping. Server never boots, no ping. All four failure modes above collapse into one signal: the ping didn’t arrive.

One caveat, and it’s the same trap as the Kubernetes version of this post — a failing ExecStartPost marks the whole unit failed, even though the real work succeeded. If your job isn’t safe to be marked failed, guard it:

ExecStartPost=-/usr/bin/curl -fsS --max-time 10 https://api.didmyjobrun.com/ping/YOUR-UUID

The leading - tells systemd to ignore the exit code.

The short version

  • list-timers tells you the timer fired, never whether the job worked.
  • systemctl --failed is the check you actually want, and nothing runs it for you.
  • OnFailure= covers jobs that ran and broke.
  • A heartbeat covers jobs that never ran — the ones OnFailure structurally cannot see.

That last category is the one Did My Job Run watches: add one line to ExecStartPost, and the silence becomes the alert.