Skip to content

[Bug]: Telegram webhook mode acks 200 before persistence — updates lost forever on crash/restart #98777

Description

@obviyus

Bug type

Behavior bug (incorrect output/state without crash)

Summary

Telegram webhook mode acks 200 before any persistence, so a gateway crash/restart mid-processing silently and permanently loses the update — Telegram saw the 200 and never redelivers.

What happened

extensions/telegram/src/webhook.ts:357-361 (at eb417fa206e) responds 200 then fire-and-forgets bot.handleUpdate(...); the failure path only logs "webhook update processing failed after ack". Polling mode durably spools every update to the SQLite ingress queue before processing, with claims/retry/dead-letter; webhook mode has no persistence at all between the ack and completion. A deploy, openclaw gateway restart, OOM, or any transient handleUpdate failure (DB lock, provider error) during an in-flight webhook turn drops the message with only a log line — the same failure on the polling path is retried durably. The early ack itself is a pinned contract (webhook.test.ts, avoids Telegram's delivery timeout); the gap is the missing durable write before it.

Directly associated webhook-mode gaps (same subsystem, second-class vs polling):

  1. No retry on startup getMe (webhook.ts:278-282): one transient network error during bot.init() fails the whole account start; polling retries recoverable setup errors indefinitely with backoff. Each failed start burns one of the supervisor's 10 cumulative restart attempts, so a reboot before DNS is up can permanently disable a webhook account.
  2. Bot transport never closed (bot-core.ts:120-124, shutdown webhook.ts:413-424): webhook mode lets the bot build its own undici transport and never calls transport.close() — the socket-leak class fix(telegram): release undici dispatcher sockets on polling transport rebuild #68128 fixed for polling only. Every webhook account restart leaks a keep-alive dispatcher pool. Related: probeTransportCache eviction (probe.ts:86-93) drops the oldest transport without closing it.
  3. Generic 120 req/min per-IP rate limit throttles Telegram's own delivery (webhook.ts:283-287, defaults src/plugin-sdk/webhook-memory-guards.ts:36-40): Telegram delivers from few source IPs with connection reuse, so a busy bot exceeding ~2 updates/sec gets 429'd on fully authenticated requests → Telegram backs off, pending_update_count grows, old updates eventually drop. The pre-auth ordering is deliberate (secret-guess budgeting), but valid and attacker traffic share one budget.

Expected behavior

Webhook updates are durably persisted (reusing the existing ingress spool) before the 200 ack and drained with the same retry/dead-letter semantics as polling; startup init retries transient errors with backoff; the transport is closed on shutdown; authenticated Telegram traffic is not rate-limited by the secret-guess budget.

Evidence

  • extensions/telegram/src/webhook.ts:357-361,379 (ack-then-forget), :278-282 (no init retry), :283-287,313-322 (pre-auth rate limit), :413-424,493-501 (shutdown paths without transport close)
  • Polling-side contrast: durable spool + drain in polling-session.ts / telegram-ingress-spool.ts; setup retry loop polling-session.ts:479-489,524-554; transport dispose polling-session.ts:445-450 (fix for fix(telegram): release undici dispatcher sockets on polling transport rebuild #68128)

Version

main @ eb417fa206e

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.maintainerMaintainer-authored PR

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions