Cron Job Monitoring: How to Detect Silent Failures in Background Jobs and Workers

A failed website request usually leaves evidence. A failed cron job may leave nothing at all.

A backup can stop running after a server migration. A billing task can exit before processing any records. A queue worker can remain alive while making no progress. Unless somebody checks the output, these failures may remain hidden for hours or days.

Heartbeat monitoring detects that silence. The job or worker sends a small HTTP request when it reaches a meaningful point, and the monitoring service opens an incident when the expected signal does not arrive.

Why normal uptime checks miss background jobs

HTTP and port monitors work by calling a target. That model fits websites and APIs, but many background workloads do not expose an endpoint:

Database backups.
Billing and invoice generation.
Data imports and exports.
Scheduled reports.
Certificate renewal tasks.
Cleanup and retention jobs.
Queue consumers and event processors.
Long-running daemons on private networks.

A server can respond normally while one of these tasks is broken. Process monitoring also has a blind spot: a worker may still be running but stuck, disconnected from its queue, or repeatedly failing the same item.

A heartbeat proves recent progress rather than simple process existence.

How heartbeat monitoring works

Heartbeat monitoring reverses the usual check direction:

Create a monitor and receive a unique URL.
Define how often the workload should report.
Call the URL from the job or worker.
Allow a small grace period for normal variation.
Open an incident when the expected ping is late.

This works well for private systems because the monitored workload makes an outbound request. You do not need to expose an inbound health endpoint or open a firewall port.

The important design decision is where to send the ping.

Ping only after useful work completes

For a scheduled job, send the success heartbeat after the job has completed and verified its result.

Do not ping at startup if the monitor is meant to prove completion. A backup script that pings before creating the backup will remain green even if the database dump fails a second later.

The success point should match the outcome you care about:

A backup file was created, verified, and copied to storage.
An import processed the expected input and committed its records.
A report was generated and delivered.
A billing run completed without unresolved failures.
A cleanup job removed or archived eligible data.

For continuous workers, ping after a successful loop or after processing recent work. If the queue can legitimately be empty, the worker can report after checking the queue and confirming its connection is healthy.

Do not tie a heartbeat to meaningless activity such as a timer firing. It should prove that the workload can still perform its job.

Set the expected interval and grace period

Two values control when a missing heartbeat becomes an incident:

Expected interval: How often the workload should report.
Grace period: Extra time allowed for scheduling jitter and normal runtime variation.

For a job that starts every hour and normally takes four to seven minutes, a reasonable deadline might be one hour plus a ten-minute grace period. A one-minute grace period would create noise during ordinary slow runs. A one-hour grace period would delay a real alert unnecessarily.

Choose the grace period from observed runtimes, not guesswork. Review several weeks of data and account for:

Normal high-percentile duration.
Scheduler startup delays.
Expected input-size variation.
Lock contention with other jobs.
Slow but acceptable dependencies.
Clock and timezone behavior.

Keep the interval and grace period separate. Extending the expected interval to hide jitter makes the configuration harder to understand and can delay detection.

For calendar schedules, confirm the timezone and daylight-saving behavior. A job expected at 02:30 local time may not run on the spring clock change, and it may run twice during the autumn change. UTC is often simpler, but the correct choice depends on the business schedule.

Detect jobs that run too long

A completion-only heartbeat detects a hung job after its final deadline. For a daily job with a generous grace period, that may be later than the team wants.

There are two common ways to detect excessive duration:

Send start and completion signals

If the monitoring platform supports them, send a start signal when execution begins and a success or failure signal when it ends. The monitor can alert when completion does not follow within the maximum runtime.

This separates three states:

The job never started.
The job started but did not finish.
The job finished unsuccessfully.

Healthchecks.io documents this pattern in its check configuration guide, where the grace time can also limit the gap between start and success signals.

Report duration as a metric

Measure runtime inside the job and attach it to the successful heartbeat. This supports trend detection even when the platform uses success-only pings.

An increase from 5 minutes to 18 minutes may reveal growing input, slower storage, database contention, or a dependency regression before the job starts missing its deadline.

Signal known failures immediately

If the job knows it failed, do not always wait for the missing-heartbeat deadline. Send an explicit failure signal when the monitoring platform supports one, then exit with the original nonzero status.

An active failure signal is useful for:

An exception or nonzero exit code.
Invalid output.
A partial import that exceeds an allowed error count.
A backup that cannot be restored or verified.
A job that processed zero records when work was expected.

Healthchecks.io, for example, supports explicit failure signals using a failure URL or exit status. The exact API differs by provider.

Missing and explicit failures carry different information. A missing ping may mean the scheduler never started, the host failed, the process hung, or the network call could not leave the environment. An explicit failure proves the job ran far enough to report its error.

Track each worker with a stable source ID

A single heartbeat URL can represent one job, but worker pools and service clusters need per-instance visibility.

Give each source a stable identifier such as:

Hostname.
Cloud instance ID.
Kubernetes pod identity when its lifecycle is meaningful.
Worker name or shard ID.
Region and availability zone.

Avoid random IDs that change on every ping. They make one worker look like many active workers and prevent useful last-seen history.

Then define the healthy range:

Minimum active sources: The smallest number of workers required to handle the workload.
Maximum active sources: The expected upper bound when duplicates or runaway scaling are dangerous.

A four-node worker pool might remain healthy with three active nodes, warn when only two remain, and fail below two. An unexpected fifth node might indicate a duplicate deployment processing the same jobs twice.

Quorum should reflect capacity and correctness, not merely the desired replica count. Losing one of three workers may be harmless for a low-volume queue but urgent during peak billing.

Attach small, useful metrics

A heartbeat can carry enough context to make the first investigation faster. Useful values include:

Runtime in seconds.
Records read, processed, skipped, and failed.
Bytes backed up or exported.
Queue depth or oldest message age.
Timestamp of the newest source record.
Output file size.
Remaining disk space.

Keep payloads small and consistent. Do not attach complete logs, database rows, access tokens, customer data, or unbounded error text.

Metrics should help answer whether the job did useful work and whether its behavior is changing. A successful import that suddenly processes zero records may deserve an alert even though the script exited cleanly.

Make the heartbeat call reliable without hiding failures

The monitoring request can fail even when the job succeeds. Treat that network call as a small but important delivery step:

Use HTTPS.
Set a short connection and request timeout.
Retry a few times with backoff.
Do not let retries extend beyond the job's operational deadline.
Log delivery failure locally.
Preserve the job's original exit status.

Do not report success from a separate timer or unrelated process. That can keep the monitor green after the actual workload stops.

For especially critical jobs, write the result to a durable local or queue-backed outbox and deliver it asynchronously. This reduces false alerts during a short network interruption, but the design must not turn an old result into a fresh heartbeat.

A simple completion heartbeat

The following shell example reports only after a backup succeeds and includes duration and output size. The heartbeat URL should come from protected configuration.

#!/usr/bin/env bash
set -euo pipefail

started_at=$(date +%s)
output_file="/backups/app-$(date +%F).dump"

/usr/local/bin/create-backup "$output_file"
/usr/local/bin/verify-backup "$output_file"

duration_seconds=$(( $(date +%s) - started_at ))
size_bytes=$(wc -c < "$output_file")

curl --fail --silent --show-error --retry 3 \
  --max-time 10 \
  -X POST "$HEARTBEAT_URL?source=$HOSTNAME" \
  -H "Content-Type: application/json" \
  -d "{\"duration_seconds\":$duration_seconds,\"size_bytes\":$size_bytes}"

Because set -e stops the script on backup or verification failure, the success ping is never sent. If the provider supports an explicit failure endpoint, add an error trap that reports the failure while retaining the original exit code.

Route alerts by impact

Not every missed job needs to page someone immediately.

Page for failures that threaten recovery, revenue, security, or a time-sensitive customer promise. Create a lower-priority ticket for a delayed internal report or cleanup task with available catch-up time.

Every alert should include:

Job or worker name and owner.
Expected schedule and grace period.
Last successful ping and source.
Known failure or missing-heartbeat state.
Recent duration and workload metrics.
Link to logs and a recovery runbook.
Whether rerunning the job is safe.

That final point matters. Some tasks are idempotent; others can duplicate charges, messages, or state changes. Document the recovery procedure before the first incident.

Common heartbeat monitoring mistakes

Pinging at job start only: Proves scheduling, not completion.
Using one monitor for unrelated jobs: Hides which workload missed its deadline.
Choosing an enormous grace period: Reduces noise by delaying real detection.
Ignoring runtime trends: Misses degradation before the deadline is breached.
Reusing one source ID for a cluster: Conceals failed workers and duplicate instances.
Treating process existence as progress: A live but stuck worker can still report healthy.
Sending sensitive logs in the ping: Turns monitoring data into a security risk.
Alerting without a safe rerun procedure: Slows recovery and risks duplicate work.

A practical setup checklist

Make silent workloads observable

Heartbeat monitoring is simple because it watches for expected evidence of progress. A scheduled job reports after completing useful work. A continuous worker reports after a healthy loop. A cluster identifies each source so the monitor can evaluate capacity instead of one shared pulse.

Testable heartbeat monitoring supports expected frequency, acceptable delay, per-source tracking, minimum and maximum active sources, and JSON metric capture. See the heartbeat monitor documentation for configuration and examples.

For related monitoring design, read our guides to multi-region checks and alert thresholds and synthetic monitoring, RUM, and APM.

Why normal uptime checks miss background jobs​

How heartbeat monitoring works​

Ping only after useful work completes​

Set the expected interval and grace period​

Detect jobs that run too long​

Send start and completion signals​

Report duration as a metric​

Signal known failures immediately​

Track each worker with a stable source ID​

Attach small, useful metrics​

Make the heartbeat call reliable without hiding failures​

A simple completion heartbeat​

Route alerts by impact​

Common heartbeat monitoring mistakes​

A practical setup checklist​

Make silent workloads observable​