The Cron Job That Failed for Six Days and Told Nobody

It is 9:15 on a Monday morning. A teammate asks why the weekly report is still showing last Tuesday's numbers. You open the server, and there it is: the cron job that builds the report has not worked in six days. Nobody noticed, because nothing ever told anyone. No error email, no log line, no alert. Just silence.

This is the most common way scheduled jobs fail. They do not crash loudly; they quietly stop, or quietly overlap, or quietly write their errors into a void. The good news is that a handful of small habits on a plain Linux server will make most of these failures visible. None of them need extra software.

A cron job that fails silently is worse than one that was never written. At least with no job, nobody trusts the result.

Why cron fails quietly

Cron runs your command in a stripped-down environment. It has a minimal PATH, no interactive shell profile, and no terminal. A script that works perfectly when you run it by hand may fail under cron simply because python3 or mysqldump is not found, or because a relative path points somewhere unexpected.

On top of that, cron's default way of reporting problems is to email the job's output to the local user. On most modern servers no mail system is configured, so that output goes nowhere. The result is a job that fails and tells nobody.

So the first principle is simple: never rely on cron to tell you what happened. Make the job tell you.

Step 1: Give every job a log file

The cheapest improvement is redirecting both standard output and standard error into a log file with timestamps. Here is a crontab entry that does it:

# m h dom mon dow  command
15 2 * * * /usr/local/bin/nightly_report.sh >> /var/log/nightly_report.log 2>&1

The >> appends output, and 2>&1 sends errors to the same place. Now there is at least a record. But raw output without times is hard to read, so add a tiny wrapper inside the script itself:

#!/usr/bin/env bash
set -Eeuo pipefail

log() { printf '%s %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; }

trap 'log "FAILED at line $LINENO (exit $?)"' ERR

log "start"
# ... real work here ...
log "done"

Three things are happening in the first lines. set -e stops the script at the first failing command, set -u treats unset variables as errors (this catches typos), and pipefail makes a pipeline fail if any stage fails, not just the last one. The trap ... ERR line writes a message with the exact line number when something breaks.

Without these, a script can hit an error on line 5, ignore it, and cheerfully print "done" on line 40.

Step 2: Set the environment explicitly

Because cron's environment is bare, state what you need at the top of the crontab or script:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=""

15 2 * * * cd /srv/reports && ./nightly_report.sh >> /var/log/nightly_report.log 2>&1

Two details matter here. Using cd to a known directory removes the "works only from my home folder" class of bugs. And always use absolute paths for anything you call: /usr/bin/python3 is more predictable than python3.

If your job is Python, call the interpreter from a specific virtual environment rather than the system one:

15 2 * * * /srv/reports/venv/bin/python /srv/reports/build_report.py >> /var/log/nightly_report.log 2>&1

That keeps a system package upgrade from changing your job's behavior overnight.

Step 3: Stop jobs from overlapping with flock

Here is a scenario that catches people. A job normally takes 4 minutes and runs every 5. One night the database is slow, the job takes 9 minutes, and now two copies run at once, both writing to the same table or file. Data gets duplicated or corrupted, and the load makes the next run slower still.

The fix is flock, which ships with most Linux distributions:

*/5 * * * * /usr/bin/flock -n /var/lock/sync_orders.lock /usr/local/bin/sync_orders.sh >> /var/log/sync_orders.log 2>&1

With -n, a second copy that finds the lock held exits immediately instead of waiting. The lock is released automatically when the process ends, even if it crashes, so you never get a stale lock file blocking everything forever. That last property is why flock beats the hand-rolled "check if a pid file exists" approach.

If you would rather know when a run was skipped, wrap it so the skip is logged:

*/5 * * * * /usr/bin/flock -n /var/lock/sync_orders.lock /usr/local/bin/sync_orders.sh >> /var/log/sync_orders.log 2>&1 || echo "$(date) skipped: previous run still active" >> /var/log/sync_orders.log

Step 4: Send a heartbeat so silence becomes an alarm

Logs only help if someone reads them. What you really want is for a missing run to raise a flag. The trick is a heartbeat: the job touches a file or pings a URL at the end of a successful run, and a separate check complains if the signal is too old.

The simplest version needs only a timestamp file:

# last line of the successful job
date +%s > /var/run/nightly_report.last_ok

And a second cron entry that runs every morning and checks the file's age:

#!/usr/bin/env bash
FILE=/var/run/nightly_report.last_ok
MAX_AGE=$((26 * 3600))   # 26 hours: daily job plus some slack

now=$(date +%s)
last=$(cat "$FILE" 2>/dev/null || echo 0)

if (( now - last > MAX_AGE )); then
  echo "nightly_report has not succeeded in over 26 hours" | logger -t cron-watch -p user.err
  # add: send to chat webhook, email, or an uptime-monitoring ping
fi

Notice the 26-hour limit rather than 24. A job that runs daily but occasionally takes a few minutes longer should not trigger false alarms, because an alarm that cries wolf gets muted within a week.

Many teams replace the file with a "dead man's switch" monitoring service: the job makes an HTTP request on success (curl -fsS --retry 3 https://your-monitor.example/ping/abc123), and the service alerts you if the ping stops arriving. The idea is identical; only the delivery changes.

Step 5: Use syslog so the system keeps your logs tidy

Writing to /var/log/yourjob.log works, but logs grow forever unless you rotate them. Two good options exist. First, you can send messages to the system logger with logger, which feeds journald or syslog and inherits their rotation:

/usr/local/bin/nightly_report.sh 2>&1 | logger -t nightly_report

Then read them with journalctl -t nightly_report --since today.

Second, if you prefer plain files, add a small logrotate rule:

# /etc/logrotate.d/nightly_report
/var/log/nightly_report.log {
    weekly
    rotate 8
    compress
    missingok
    notifempty
}

Eight compressed weeks of history is usually plenty, and costs almost no disk space.

A quick checklist before you trust any scheduled job

Run through this table once for each job on your server. Most people find at least two gaps on the very first pass.

The short version

Cron is reliable at starting commands and terrible at telling you how they went. Fix that by logging with timestamps, making the script fail loudly, pinning the environment, preventing overlaps with flock, and adding a heartbeat so that absence triggers an alert.

An hour spent on these habits will save you from the Monday-morning surprise. And the next time a teammate asks why the report looks stale, you will already know, because your server told you first.