-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[Bug]: Telegram webhook mode acks 200 before persistence — updates lost forever on crash/restart #98777
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maintainerMaintainer-authored PRMaintainer-authored PR
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maintainerMaintainer-authored PRMaintainer-authored PR
Type
Fields
Priority
None yet
Bug type
Behavior bug (incorrect output/state without crash)
Summary
Telegram webhook mode acks 200 before any persistence, so a gateway crash/restart mid-processing silently and permanently loses the update — Telegram saw the 200 and never redelivers.
What happened
extensions/telegram/src/webhook.ts:357-361(ateb417fa206e) responds 200 then fire-and-forgetsbot.handleUpdate(...); the failure path only logs "webhook update processing failed after ack". Polling mode durably spools every update to the SQLite ingress queue before processing, with claims/retry/dead-letter; webhook mode has no persistence at all between the ack and completion. A deploy,openclaw gateway restart, OOM, or any transient handleUpdate failure (DB lock, provider error) during an in-flight webhook turn drops the message with only a log line — the same failure on the polling path is retried durably. The early ack itself is a pinned contract (webhook.test.ts, avoids Telegram's delivery timeout); the gap is the missing durable write before it.Directly associated webhook-mode gaps (same subsystem, second-class vs polling):
getMe(webhook.ts:278-282): one transient network error duringbot.init()fails the whole account start; polling retries recoverable setup errors indefinitely with backoff. Each failed start burns one of the supervisor's 10 cumulative restart attempts, so a reboot before DNS is up can permanently disable a webhook account.bot-core.ts:120-124, shutdownwebhook.ts:413-424): webhook mode lets the bot build its own undici transport and never callstransport.close()— the socket-leak class fix(telegram): release undici dispatcher sockets on polling transport rebuild #68128 fixed for polling only. Every webhook account restart leaks a keep-alive dispatcher pool. Related:probeTransportCacheeviction (probe.ts:86-93) drops the oldest transport without closing it.webhook.ts:283-287, defaultssrc/plugin-sdk/webhook-memory-guards.ts:36-40): Telegram delivers from few source IPs with connection reuse, so a busy bot exceeding ~2 updates/sec gets 429'd on fully authenticated requests → Telegram backs off,pending_update_countgrows, old updates eventually drop. The pre-auth ordering is deliberate (secret-guess budgeting), but valid and attacker traffic share one budget.Expected behavior
Webhook updates are durably persisted (reusing the existing ingress spool) before the 200 ack and drained with the same retry/dead-letter semantics as polling; startup init retries transient errors with backoff; the transport is closed on shutdown; authenticated Telegram traffic is not rate-limited by the secret-guess budget.
Evidence
extensions/telegram/src/webhook.ts:357-361,379(ack-then-forget),:278-282(no init retry),:283-287,313-322(pre-auth rate limit),:413-424,493-501(shutdown paths without transport close)polling-session.ts/telegram-ingress-spool.ts; setup retry looppolling-session.ts:479-489,524-554; transport disposepolling-session.ts:445-450(fix for fix(telegram): release undici dispatcher sockets on polling transport rebuild #68128)Version
main @
eb417fa206e