What Problem This Solves

Fixes an issue where a cron job mutation that fails to persist still changes the live scheduler, so what the user was told no longer matches what the gateway does or what survives a restart.

cron add, cron update, and cron remove mutate the in-memory cron store first and persist to SQLite afterwards. When the persist fails (e.g. disk full or a failed write transaction), the caller correctly receives an error — but the in-memory mutation stays behind in the running gateway:

• add: a job the user was told failed to save stays live, can be armed and executed, and is even written to disk by the next unrelated successful persist. If the gateway restarts first, it silently vanishes instead.
• update: the live scheduler runs the new version the user was told did not apply, and a restart silently reverts it to the old version.
• remove: the job disappears from the live scheduler but resurrects from SQLite on the next restart.

Both user surfaces converge on the same code path — the CLI issues the cron.add/cron.update/cron.remove gateway RPCs (src/cli/cron-cli.ts), whose handlers call CronService.add/update/remove (src/cron/service.ts), which delegate to the ops in this diff. The in-memory store is trusted indefinitely once loaded (ensureLoaded fast path), so none of this self-heals while the process lives.

Why This Change Was Made

The narrowest fix that restores the invariant "if the caller was told the mutation failed, the runtime behaves as if it never happened": snapshot the mutable state before the mutation and restore it when the persist throws, then rethrow the original error unchanged.

The snapshot covers the whole store plus the catch-up deferral marker set, not just the touched job, because recomputeNextRunsForMaintenance runs inside these operations and mutates nextRunAtMs/deferral state across all jobs — restoring only the touched job would leave sibling scheduler state ahead of disk. This matches the persist-before-mutate durability convention already documented in the task registry (src/tasks/task-registry.ts), which the cron service predates.

Non-goal: persist() has a sibling seam where a failed quarantine flush silently skips the save (src/cron/service/store.ts); fixing it changes error propagation for read-path callers that intentionally degrade and continue, so it is left to a separately scoped follow-up. Run-state bookkeeping persists (prepareManualRun, finishPreparedManualRun) are also untouched — the run already happened, so rollback semantics differ.

User Impact

Operators get truthful outcomes from cron CRUD under storage failure: a failed cron add/update/remove now restores the cron store and catch-up deferral markers touched by these operations, matching both the error the user saw and the state the next restart loads. No phantom runs of jobs that "failed to save", no silent reverts of edits, no deleted jobs resurrecting after a restart. Error reporting and the success paths are unchanged.

Evidence

Real behavior proof (before/after, real CronService + real shared SQLite state DB)

A three-phase driver script runs the real CronService facade against the real shared SQLite state DB (OPENCLAW_STATE_DIR pointed at a scratch dir; separate OS processes per phase; no JS mocks or module patching). The durable-write failure is a genuine SQLITE_READONLY error raised inside saveCronJobsStore's real write transaction, by switching the live production connection to read-only state (PRAGMA query_only=1 on the same cached handle the cron service persists through) between a successful seed and the mutations.

Before (PR base, upstream/main): every failed mutation leaves the live scheduler lying, and a restart flips all three back:

seed: added job "daily-report"
  live store after seed: 1 job(s): daily-report

SQLite state DB is now read-only (PRAGMA query_only=1 on the live connection)

--- cron add (durable write fails) ---
  add FAILED (real SQLite error): attempt to write a readonly database
  live store after failed add: 2 job(s): daily-report, phantom-job      <- phantom job live
--- cron update: rename daily-report -> renamed-report (durable write fails) ---
  update FAILED (real SQLite error): attempt to write a readonly database
  live store after failed update: 2 job(s): renamed-report, phantom-job <- rejected rename live
--- cron remove daily-report (durable write fails) ---
  remove FAILED (real SQLite error): attempt to write a readonly database
  live store after failed remove: 1 job(s): phantom-job                 <- real job gone from live

fresh process (simulated gateway restart), SQLite writable again:
  store loaded from SQLite: 1 job(s): daily-report                      <- restart contradicts everything above

After (this PR, same script, same failure): the live store stays consistent with the reported errors, and restart agrees:

SQLite state DB is now read-only (PRAGMA query_only=1 on the live connection)

--- cron add (durable write fails) ---
  add FAILED (real SQLite error): attempt to write a readonly database
  live store after failed add: 1 job(s): daily-report
--- cron update: rename daily-report -> renamed-report (durable write fails) ---
  update FAILED (real SQLite error): attempt to write a readonly database
  live store after failed update: 1 job(s): daily-report
--- cron remove daily-report (durable write fails) ---
  remove FAILED (real SQLite error): attempt to write a readonly database
  live store after failed remove: 1 job(s): daily-report

fresh process (simulated gateway restart), SQLite writable again:
  store loaded from SQLite: 1 job(s): daily-report

Transcripts are from a Linux host, redacted only by using scratch paths. The driver script:

import fs from "node:fs";
import path from "node:path";
import { CronService } from "./src/cron/service.js";
import { openOpenClawStateDatabase } from "./src/state/openclaw-state-db.js";

const stateDir = process.env.OPENCLAW_STATE_DIR!;
const phase = process.argv[2];
const storePath = path.join(stateDir, "cron", "jobs.json");

const log = {
  debug: () => {},
  info: () => {},
  warn: (_o: unknown, m?: string) => console.log(`  [warn] ${m ?? ""}`),
  error: (_o: unknown, m?: string) => console.log(`  [error] ${m ?? ""}`),
};

function makeService() {
  return new CronService({
    log,
    storePath,
    cronEnabled: false, // scheduler timers off; CRUD + persistence paths only
    enqueueSystemEvent: () => {},
    requestHeartbeat: () => {},
    runIsolatedAgentJob: async () => ({ status: "ok" as const }),
  });
}

function jobInput(name: string) {
  return {
    name,
    enabled: true,
    schedule: { kind: "cron", expr: "0 0 * * *" } as const,
    sessionTarget: "isolated" as const,
    wakeMode: "next-heartbeat" as const,
    payload: { kind: "agentTurn", message: `run ${name}` } as const,
  };
}

async function show(svc: CronService, label: string) {
  const jobs = await svc.list({ includeDisabled: true });
  console.log(`  ${label}: ${jobs.length} job(s): ${jobs.map((j) => `${j.name}`).join(", ") || "(none)"}`);
  return jobs;
}

const svc = makeService();

if (phase === "seed") {
  const job = await svc.add(jobInput("daily-report"));
  console.log(`seed: added job "daily-report" id=${job.id}`);
  await show(svc, "live store after seed");
  fs.writeFileSync(path.join(stateDir, "job-id.txt"), job.id);
} else if (phase === "mutate") {
  const jobId = fs.readFileSync(path.join(stateDir, "job-id.txt"), "utf-8").trim();
  await show(svc, "live store loaded from SQLite at process start");

  // Put the shared production SQLite connection (the same cached handle the
  // cron service persists through) into a read-only state. Every durable
  // write below fails with a genuine SQLITE_READONLY error raised inside
  // saveCronJobsStore's real write transaction — no JS mocking involved.
  const { db } = openOpenClawStateDatabase();
  db.exec("PRAGMA query_only = 1");
  console.log("\nSQLite state DB is now read-only (PRAGMA query_only=1 on the live connection)");

  console.log("\n--- cron add (durable write fails) ---");
  try {
    await svc.add(jobInput("phantom-job"));
    console.log("  add unexpectedly succeeded");
  } catch (err) {
    console.log(`  add FAILED (real SQLite error): ${(err as Error).message}`);
  }
  await show(svc, "live store after failed add");

  console.log("\n--- cron update: rename daily-report -> renamed-report (durable write fails) ---");
  try {
    await svc.update(jobId, { name: "renamed-report" });
    console.log("  update unexpectedly succeeded");
  } catch (err) {
    console.log(`  update FAILED (real SQLite error): ${(err as Error).message}`);
  }
  await show(svc, "live store after failed update");

  console.log("\n--- cron remove daily-report (durable write fails) ---");
  try {
    await svc.remove(jobId);
    console.log("  remove unexpectedly succeeded");
  } catch (err) {
    console.log(`  remove FAILED (real SQLite error): ${(err as Error).message}`);
  }
  await show(svc, "live store after failed remove");

  db.exec("PRAGMA query_only = 0");
} else if (phase === "inspect") {
  console.log("fresh process (simulated gateway restart), SQLite writable again:");
  await show(svc, "store loaded from SQLite");
} else {
  throw new Error(`unknown phase: ${phase}`);
}

Failing-first tests

src/cron/service/ops.test.ts (cron service ops persist rollback describe, persist failure injected via saveCronJobsStore rejection): before the fix all five fail with the exact lying behaviors above — notably the recovery case shows the phantom job from a failed add being written to disk by the next successful persist:

FAIL > recovers after a failed persist so the next mutation succeeds
- Expected  + Received
  [
+   "f97b06c3-...",   <- job whose add reported "disk full" to the caller
    "9868ddb1-...",
  ]

After the fix:

✓ rolls back an added job from the live store when persist fails
✓ keeps the pre-update job in the live store when persist fails
✓ keeps a removed job in the live store when persist fails
✓ keeps a job's catch-up deferral marker when a remove persist fails
✓ recovers after a failed persist so the next mutation succeeds

Test Files  1 passed (1) — src/cron/service/ops.test.ts, 24 tests

Full cron suite (local, node scripts/run-vitest.mjs src/cron/, ~34s): 117 test files / 1237 tests pass. Production diff is confined to src/cron/service/ops.ts (snapshot + restore helpers, three call sites); no change to success-path behavior, event ordering, error types, or --json shapes.