Skip to content

cron: startup overflow catch-up deferral dropped by cron list/status (any maintenance recompute clobbers it) #93935

Description

@yetval

Summary

A cron job whose startup catch-up overflows the per-restart budget (more than maxMissedJobsPerRestart non-agentTurn jobs overdue after gateway downtime) is deferred to a near-future staggered catch-up slot (now + stagger). That deferral is silently advanced to the job's next natural schedule slot (for a daily 0 9 * * * job, tomorrow 09:00) the moment any read RPC runs (cron list/status, a UI or Mac-app poll), or the first overflow job finalizes, or an empty-due timer tick fires. The deferred missed run never executes; it is dropped for a full period.

Environment

  • Commit: cc451f9 (origin/main at time of filing)
  • Surface: cron service scheduler (src/cron/service/*); any caller of recomputeNextRunsForMaintenance other than start()'s post-catchup pass

Steps to reproduce

  1. Have more than maxMissedJobsPerRestart (default 5) non-agentTurn cron jobs overdue while the gateway is down (here: 5 hourly 0 * * * * jobs plus 1 daily 0 9 * * * job).
  2. Start the cron service. runMissedJobs runs the first 5 and defers the daily overflow to now + stagger (startNow + 5000). start()'s maintenance pass correctly preserves it (this is what fix(cron): preserve startup overflow catch-up deferrals in start() maintenance pass #93810 fixed).
  3. Before the staggered catch-up tick fires, issue a plain read RPC: cron list followed by cron status.
  4. Observe the daily job's nextRunAtMs.

Expected

The deferred catch-up slot (startNow + 5000) survives the read RPC and fires on the staggered tick.

Actual

The read RPC advances the daily job's nextRunAtMs to its natural slot (tomorrow 09:00). The missed run is dropped.

Root cause

#93810 scoped its exemption to a single local skipFutureRepairJobIds set threaded only into start()'s second maintenance pass:

src/cron/service/ops.ts:269

const changed = recomputeNextRunsForMaintenance(state, {
  recomputeExpired: true,
  skipFutureRepairJobIds: deferredCatchupJobIds,
});

The deferral is not recorded anywhere durable, so every other caller of recomputeNextRunsForMaintenance(state) runs with the default-enabled future-slot repair and no skip set:

  • ensureLoadedForRead (src/cron/service/ops.ts:210) -> invoked by status/list/readJob/listPage
  • finalizeCompletedResults (src/cron/service/timer.ts:1297) -> after the first overflow job fires, clobbers the still-deferred remainder
  • the empty-due maintenance pass (src/cron/service/timer.ts:1183)

For an overflow deferral at now + stagger, shouldRepairFutureCronNextRunAtMs (src/cron/service/jobs.ts:197) returns true: nextRun < naturalNext, payload is non-agentTurn, and the slot is not a recognized staggered cron slot. The job is advanced to its natural slot.

Sibling surfaces

Real behavior proof

Behavior addressed: a startup overflow catch-up deferral is dropped when a maintenance recompute (read RPC / finalize / empty-due tick) runs before its staggered tick.
Real environment tested: real CronService ops driven end to end (start() then list() + status()) at commit cc451f9, via src/cron/service.overflow-readclobber.repro.test.ts.
Exact steps or command run after this patch: node scripts/run-vitest.mjs src/cron/service.overflow-readclobber.repro.test.ts
Evidence after fix:

# OBSERVED (buggy, origin/main cc451f98cb): a plain cron list/status drops the deferral
startNow=1765645200000
deferralExpected(startNow+5000)=1765645205000
naturalNextDaily(2025-12-14T09:00Z)=1765702800000
dailySlot.afterStart=1765645205000        # #93810 preserves it through start()
dailySlot.afterReadRPCs=1765702800000     # clobbered to tomorrow 09:00 by list()+status()
clobberedToNaturalSlot=true

# EXPECTED (correct behavior on identical inputs): deferral survives the read RPC
dailySlot.afterStart=1765645205000
dailySlot.afterReadRPCs=1765645205000     # still the staggered catch-up slot

Observed result after fix: on origin/main the daily overflow deferral is advanced from the staggered catch-up slot (startNow + 5000) to the next natural slot (tomorrow 09:00) by a read-only cron list/status, dropping the missed run; the correct behavior keeps the staggered slot until it fires.
What was not tested: the finalizeCompletedResults and empty-due-tick sibling paths were not separately driven end to end (they are enumerated by identical call site, not independently reproduced); not run against a live multi-process gateway, only the in-process CronService.

Suggested fix

Persist the exemption on the in-memory service state instead of a one-shot local. Add state.pendingCatchupDeferralJobIds; populate it where deferrals are assigned (ops.start + timer.applyStartupCatchupOutcomes); have recomputeNextRunsForMaintenance always exempt those ids and clear each id once its slot is reached (now >= nextRunAtMs). This replaces and deletes the now-redundant skipFutureRepairJobIds option, giving one canonical path that covers all recompute callers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions