Why can a Kubernetes CronJob run twice, overlap, or skip?

Some customers received the same invoice email twice. The chain behind it is simple. A CronJob creates a Job on schedule, and that Job creates Pods, retrying them if needed. Those Pods call the Email API, which delivers to the customer. Duplicates can start at either layer, the CronJob or the Job.

For the demo, the billing CronJob fires every 10 minutes, but one run takes 15. At 00:00 the controller creates Job 00:00, and it starts sending invoices through the Email API. At 00:10, the default policy, Allow, creates Job 00:10 anyway. It loops over the same unpaid invoices, so customers get duplicates. Job 00:00 only finishes at 00:15.

At 00:10, the old run is still going, so concurrencyPolicy decides what happens. Allow starts a second Job, duplicating sends. Forbid skips the new run. Replace kills the old Job mid-way, which can leave emails partly sent.

Overlap isn't the only way to get duplicates. Here, Pod A sends invoices one through forty to the Email API, then crashes. The Job sees the failure and, within backoffLimit, starts Pod B.

Pod B has no memory of progress. It starts from invoice one and resends all forty. That's one CronJob run, with no overlap at all, and customers still get duplicate emails.

Now the opposite failure. The 00:20 run is due, but the controller is down or late. When it comes back, it checks the time against startingDeadlineSeconds. Inside that window, the Job starts late. Outside it, the run is skipped. Skipped runs can also happen under Forbid, and they count as missed schedules.

A CronJob schedules attempts, not guarantees. Kubernetes runs it roughly once per schedule, so your code must make repeats harmless. That's why the docs advise idempotent work.

Three fields matter here. Forbid stops a new Job from starting while the last one runs, so the 00:10 overlap goes away. startingDeadlineSeconds allows a late start up to five minutes, otherwise the run is skipped. backoffLimit caps retries at two. But retries still repeat work, so config alone can't prevent duplicates.

Config can't make the job safe, so the script does. Before sending, it inserts the invoice id into sent_invoices. If the row already exists, nothing is claimed and the email is skipped. Overlapping Jobs or a retried Pod skip invoices 1 through 40. One gotcha: a crash after claiming, before sending, needs reconciliation.

Don't guess what ran, check the history. Two Jobs whose durations and ages overlap tell you overlap happened. In the second Job, an Error Pod next to a Completed Pod shows a retry. Then describe the CronJob, and its Events show what was created or missed.

Kubernetes runs a CronJob about once per schedule, not exactly once. Overlap starts at the CronJob layer, where a new Job appears while the old one runs. Retries start at the Job layer, where a replacement Pod resends. So use Forbid, make sends idempotent, and check job history.