fix: pace delivery recovery after startup outages by ZengWen-DT · Pull Request #101118 · openclaw/openclaw

ZengWen-DT · 2026-07-06T18:53:17Z

What Problem This Solves

Startup delivery recovery could replay a large eligible backlog in a tight sequential burst after an outage or restart. Reconnect-triggered drains could run at the same time and bypass any startup-only pacing, concentrating sends against channel and provider rate limits.

Why This Change Was Made

Outbound startup recovery and reconnect drains now share one replay pacer; session-delivery startup recovery uses the same pacing primitive with its own owner. The coordinator grants replay starts at least 250ms apart, lets the first replay start immediately, and serializes concurrent callers without holding delivery execution behind a global lock.

Pacing remains inside the existing recovery deadline. Each sleep is capped to the remaining budget, so a small budget defers the backlog tail instead of overrunning the 60-second startup ceiling. Backoff skips, max-retry moves, active-entry claims, and retry counters keep their existing semantics.

User Impact

Queued messages recovered after a restart or reconnect are replayed more gently instead of all eligible entries being sent back-to-back. This reduces avoidable rate-limit bursts while preserving the bounded startup path and reconnect behavior.

Evidence

Before: both startup recovery loops sorted eligible SQLite queue entries and immediately iterated them; reconnect drains were independently callable. There was no shared pacing owner between those entry points.

After: focused regressions cover:

outbound and session startup replay spacing;
deadline-bound tail deferral with a 1ms recovery budget;
concurrent outbound startup and reconnect drains sharing the same pacing floor;
existing reconnect selection, retry/backoff, active-claim, queue-policy, and session recovery behavior.

Local focused proof on the reviewed patch lineage:

src/infra/outbound/delivery-queue.recovery.test.ts
src/infra/outbound/delivery-queue.reconnect-drain.test.ts
src/infra/outbound/delivery-queue.policy.test.ts
src/infra/session-delivery-queue.recovery.test.ts

Test Files 4 passed
Tests 79 passed

Real SQLite queue proof observed a 252ms interval between two eligible recovered deliveries with zero failures. Fresh autoreview completed clean after addressing three findings: retain the recovery deadline, cap pacing sleep to remaining budget, and share outbound pacing with reconnect drains.

Exact-head hosted CI and the additional remote-proof disposition are recorded in the maintainer verification comment before merge.

AI-assisted: implementation and review were AI-assisted; the evidence above was independently exercised against the queue and test surfaces.

clawsweeper · 2026-07-06T19:53:08Z

Codex review: needs maintainer review before merge. Reviewed July 6, 2026, 7:46 PM ET / 23:46 UTC.

Summary
The branch adds a shared 250ms startup recovery replay pacing helper and uses it in outbound and session delivery recovery, with fake-timer regression tests for spacing and budget behavior.

PR surface: Source +58, Tests +205. Total +263 across 5 files.

Reproducibility: yes. Source inspection shows current main replays eligible outbound and session recovery entries in tight sequential loops, and the PR body includes real SQLite recovery output showing after-fix spacing; I did not run tests because this review is read-only.

Review metrics: 1 noteworthy metric.

Startup replay pacing: 250ms between attempted replays; 0 config surfaces. This is the intentional message-delivery timing behavior maintainers should notice before merge.

Stored data model
Persistent data-model change detected: serialized state: src/infra/session-delivery-queue-recovery.ts, serialized state: src/infra/session-delivery-queue.recovery.test.ts, unknown-data-model-change: src/infra/session-delivery-queue.recovery.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #101058
Summary: This PR is the live candidate fix for the linked delivery recovery pacing issue; search found only adjacent delivery reliability work, not another canonical pacing fix.

Members:

canonical: [Tech Debt]: Missing exponential backoff jitter in delivery-queue recoverPendingDeliveries #101058 - The PR body uses closing syntax for this issue, and live issue metadata reports this PR as the closing pull request reference.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Risk before merge

[P0] Startup recovery now intentionally spaces every eligible replay by 250ms, so very large outage backlogs recover more slowly even though the follow-up tests show pacing sleep no longer consumes the active recovery budget.
[P1] The branch is behind current main, so branch protection or the merge queue should refresh exact-head checks before landing.

Maintainer options:

Accept fixed recovery pacing (recommended)
Merge after maintainers accept 250ms startup replay spacing as the desired rate-limit tradeoff for recovered queued messages.
Tune the pacing policy first
Adjust the spacing constant or scheduling policy if maintainers want a different recovery throughput target, then refresh focused proof.

Next step before merge

No repair job is needed; the previous budget blocker is addressed and the remaining action is normal maintainer review, rebase if required, and merge handling.

Security
Cleared: The diff only changes TypeScript recovery pacing logic and tests; it adds no dependencies, lockfile changes, workflows, permissions, downloads, or secret-handling paths.

Review details

Best possible solution:

Keep the shared startup replay pacing with the added outbound and session regression tests, and merge after maintainers accept the fixed 250ms delivery-timing tradeoff on an up-to-date head.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main replays eligible outbound and session recovery entries in tight sequential loops, and the PR body includes real SQLite recovery output showing after-fix spacing; I did not run tests because this review is read-only.

Is this the best way to solve the issue?

Yes. Applying shared pacing at the startup recovery replay point is narrower than changing retry backoff globally, and the follow-up correctly excludes pacing sleep from the active recovery budget for both affected queues.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 054e3675e19a.

Label changes

Label changes:

add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes copied terminal output from real SQLite queue recovery showing a 252ms after-fix replay gap, plus focused tests, type/lint output, CI, and a local inference smoke as supplemental proof.
remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P2: This is a bounded delivery reliability fix for startup recovery bursts with source-level reproduction and no evidence of a live P0/P1 outage.
merge-risk: 🚨 message-delivery: Merging changes how quickly queued outbound and session messages replay after startup outages, which can affect delivery timing for large backlogs.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes copied terminal output from real SQLite queue recovery showing a 252ms after-fix replay gap, plus focused tests, type/lint output, CI, and a local inference smoke as supplemental proof.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied terminal output from real SQLite queue recovery showing a 252ms after-fix replay gap, plus focused tests, type/lint output, CI, and a local inference smoke as supplemental proof.

Evidence reviewed

PR surface:

Source +58, Tests +205. Total +263 across 5 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	3	60	2	+58
Tests	2	205	0	+205
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	5	265	2	+263

What I checked:

Repository policy read: Root AGENTS.md and src/infra/outbound/AGENTS.md were read fully; the review applied the whole-surface, sibling-session-path, and message-delivery merge-risk guidance. (AGENTS.md:1, 054e3675e19a)
Current main tight replay: Current main loads pending outbound deliveries, sorts them, and calls drainQueuedEntry in the startup recovery loop without a pacing sleep between eligible entries. (src/infra/outbound/delivery-queue-recovery.ts:731, 054e3675e19a)
Sibling session replay: Current main has the same unpaced startup loop for session deliveries, so the PR's sibling-session change is a relevant one-sided-fix guard rather than unrelated churn. (src/infra/session-delivery-queue-recovery.ts:206, 054e3675e19a)
PR diff after follow-up: The latest PR head adds waitForRecoveryReplayPace, tracks pacedDelayMs separately from active recovery time, and applies the helper to both outbound and session startup recovery loops. (src/infra/delivery-recovery.shared.ts:39, a3d5e2153593)
Regression coverage added: The PR adds fake-timer coverage for 250ms replay spacing and for tiny maxRecoveryMs cases where eligible outbound and session backlog tails still drain. (src/infra/outbound/delivery-queue.recovery.test.ts:118, a3d5e2153593)
Existing helper preflight: Existing retry/backoff helpers cover failed-operation retries and jittered retry timing, while this PR needs per-entry startup replay pacing in the recovery loop; the new recovery-local hook is the narrower fix point. (src/infra/retry.ts:107, 054e3675e19a)

Likely related people:

Peter Steinberger: Recent history shows the outbound recovery split and generic delivery recovery hardening in the central files touched by this PR. (role: recent area contributor; confidence: high; commits: 562e4a1791fb, 856592cf001b; files: src/infra/outbound/delivery-queue-recovery.ts, src/infra/delivery-recovery.shared.ts)
nabbilkhan: git log -S traces the original outbound delivery crash recovery startup behavior to the delivery crash recovery commit. (role: introduced behavior; confidence: medium; commits: 207e2c5affa9; files: src/infra/outbound/delivery-queue.ts, src/infra/outbound/delivery-queue-recovery.ts)
Marcus Castro: Recent reconnect-drain work touches the same pending-delivery recovery invariant and sibling tests that this PR is intentionally leaving unchanged. (role: adjacent owner; confidence: medium; commits: 95d467398e46; files: src/infra/outbound/delivery-queue-recovery.ts, src/infra/outbound/delivery-queue.reconnect-drain.test.ts)
Agustin Rivera: Session replay history identifies recent queued session context recovery work on the sibling session-delivery path this PR also updates. (role: adjacent session recovery contributor; confidence: medium; commits: 48aae82bbc19; files: src/infra/session-delivery-queue-recovery.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (1 earlier review cycle)

reviewed 2026-07-06T20:04:47.802Z sha bcb9922 :: found issues before merge. :: [P2] Keep paced recovery from stranding large backlogs

ZengWen-DT · 2026-07-06T23:27:53Z

@clawsweeper re-review

Updated the PR to address the paced backlog tail finding. The follow-up keeps replay pacing, excludes pacing sleep from the active recovery budget, and adds outbound/session regression tests proving eligible backlog tails still drain under a tiny maxRecoveryMs.

clawsweeper · 2026-07-06T23:27:55Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

steipete · 2026-07-07T04:57:56Z

Maintainer review and pre-merge proof complete at d1f48144c179f3b00978cd954ab98979ba6136c4.

The final patch uses one shared outbound replay pacer for startup and reconnect drains, plus the same primitive under the session-delivery owner. Replay starts are spaced by 250ms, the first eligible replay remains immediate, skipped/backoff/max-retry entries do not consume a pacing grant, and every wait is capped by the existing wall-clock recovery deadline.

Proof:

Exact-head secretless hosted CI run 28841881263, attempt 2: all required jobs passed at d1f48144c17, including build, type/lint/check gates, full test shards, and QA smoke.
Attempt 1 had one unrelated TUI PTY session-adoption timeout; the same shard passed on rerun without a code change.
Focused outbound startup, reconnect, queue-policy, and session-recovery coverage passed 79/79 tests on the reviewed patch lineage using the real SQLite queue store.
Real SQLite replay proof observed 252ms between two eligible recovered deliveries, zero failures, and both entries acknowledged.
oxfmt on every changed file and git diff --check passed.
Fresh autoreview found and drove three substantive fixes: keep pacing inside maxRecoveryMs, cap sleeps to the remaining deadline, and prevent reconnect drains from bypassing startup pacing. Final full-patch autoreview was clean; no accepted/actionable findings remain.

Additional remote-proof disposition: a sanitized direct AWS Crabbox allocation was requested with public networking, Tailscale disabled, empty instance profile, and no credential hydration. The coordinator returned no lease after its full 30-minute window (silver-crayfish, pending id cbx_369ff42f06e8), and no late lease exists to stop. Per the untrusted-fork policy, exact-head secretless fork CI is the fallback proof. No live provider 429/outage call was run; the exact queue behavior is covered through SQLite recovery and the channel-independent delivery boundary.

steipete · 2026-07-07T04:58:17Z

Merged via squash.

Prepared head SHA: d1f48144c179f3b00978cd954ab98979ba6136c4
Landed commit: 78e4672d651159700987d5be224fbb16eb0819e5

* fix: pace delivery recovery replays * fix: preserve paced delivery recovery budget * fix(delivery): preserve recovery deadline while pacing * fix(delivery): coordinate recovery replay pacing --------- Co-authored-by: Peter Steinberger <steipete@gmail.com>

openclaw-barnacle Bot added the size: S label Jul 6, 2026

ZengWen-DT force-pushed the fix/101058-delivery-recovery-jitter branch from afeb7a8 to bcb9922 Compare July 6, 2026 19:00

openclaw-barnacle Bot added size: M and removed size: S labels Jul 6, 2026

steipete self-assigned this Jul 7, 2026

clawsweeper Bot mentioned this pull request Jul 7, 2026

fix(infra): add randomized jitter to delivery queue recovery #101279

Closed

ZengWen-DT and others added 4 commits July 7, 2026 05:35

fix: pace delivery recovery replays

67706d1

fix: preserve paced delivery recovery budget

3dfed0b

fix(delivery): preserve recovery deadline while pacing

14cfef6

fix(delivery): coordinate recovery replay pacing

d1f4814

steipete force-pushed the fix/101058-delivery-recovery-jitter branch from e8c9d6e to d1f4814 Compare July 7, 2026 04:36

steipete merged commit 78e4672 into openclaw:main Jul 7, 2026
169 of 170 checks passed

clawsweeper Bot mentioned this pull request Jul 7, 2026

fix: stagger delivery recovery retries #101365

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix: pace delivery recovery after startup outages#101118

fix: pace delivery recovery after startup outages#101118
steipete merged 4 commits into
openclaw:mainfrom
ZengWen-DT:fix/101058-delivery-recovery-jitter

ZengWen-DT commented Jul 6, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

ZengWen-DT commented Jul 6, 2026

Uh oh!

clawsweeper Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

ZengWen-DT commented Jul 6, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZengWen-DT commented Jul 6, 2026

Uh oh!

clawsweeper Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZengWen-DT commented Jul 6, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jul 6, 2026 •

edited

Loading

clawsweeper Bot commented Jul 6, 2026 •

edited

Loading