Skip to content

fix(gateway): recover config hot-reload after watcher errors#92027

Merged
vincentkoc merged 1 commit into
openclaw:mainfrom
hansraj316:fix/gateway-config-watcher-recovery
Jun 10, 2026
Merged

fix(gateway): recover config hot-reload after watcher errors#92027
vincentkoc merged 1 commit into
openclaw:mainfrom
hansraj316:fix/gateway-config-watcher-recovery

Conversation

@hansraj316

Copy link
Copy Markdown
Contributor

Summary

A chokidar watcher error permanently disabled config hot-reload with only a single warn. Re-create the watcher with bounded backoff (500ms/2s/5s, 3 retries); on exhausted budget escalate to log.error and flip a persistent hotReloadStatus to disabled. stop() clears any pending re-create timer.

  • Files: src/gateway/config-reload.ts
  • No CHANGELOG.md edit (release-only per AGENTS.md).

Why core (not ClawHub)

Core stability fix touching src/gateway/config-reload.ts — per AGENTS.md these stay in core (security/core hardening, bundled regressions, missing core APIs), not optional skill bundles routed to ClawHub.

Verification

  • node scripts/run-vitest.mjs run src/gateway/config-reload.test.ts => 328 passed
  • pnpm tsgo:core (tsconfig.core.json) => clean
  • oxfmt --check + oxlint on touched files => clean

Real behavior proof

  • Behavior addressed: A transient config-watcher error now recovers hot-reload via bounded backoff instead of silently disabling it forever; permanent failure is escalated and exposed via status.
  • Real environment tested: macOS (Darwin 25.3.0); OpenClaw at base 2accfee; Node 22; Vitest via scripts/run-vitest.mjs.
  • Exact steps or command run after this patch: node scripts/run-vitest.mjs run src/gateway/config-reload.test.ts
  • Evidence after fix: 328 tests pass, incl. two new cases: re-create fires only after the backoff timer (status stays active); a 4th error past the retry budget flips status to disabled and logs at error level.
  • Observed result after fix: Stale-source guard prevents double-consuming the retry budget; existing { stop } consumers unaffected (extra hotReloadStatus is structurally compatible).
  • What was not tested: Retry budget does not reset after a successful recovery (intentional, bounded); status not yet wired into an operator endpoint; full Crabbox; Linux CI.. Independently reviewed by a separate agent before commit; not yet run through Crabbox/$autoreview.

🤖 Generated with Claude Code

A chokidar watcher 'error' permanently disabled config hot-reload with a
single warn. Re-create the watcher with bounded backoff (500ms/2s/5s, 3
retries); on exhausted budget escalate to log.error and flip a persistent
hotReloadStatus to disabled. stop() clears any pending re-create timer.
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: S triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI. labels Jun 10, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ff693011de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +435 to +436
watcher = null;
void source?.close().catch(() => {});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Await the failed watcher before completing stop

When stop() runs during the retry backoff, this fire-and-forget close is the only reference to the failed watcher because watcher has already been set to null; stop() therefore resolves without waiting for Chokidar to release its filesystem handles. Chokidar documents close() as asynchronous and requiring await, so shutdown can finish while the watcher is still active, and a retry can likewise create a replacement before the old watcher has closed. Track this close promise and await it from both recreation and stop().

Useful? React with 👍 / 👎.

@vincentkoc vincentkoc self-assigned this Jun 10, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Maintainer verification complete.

  • Rebased onto current origin/main.
  • Fixed the recovery lifecycle to reset retry budget only after watcher readiness, await failed-watcher cleanup, and reconcile config changes missed during backoff.
  • Focused proof: node scripts/run-vitest.mjs src/gateway/config-reload.test.ts — 344 tests passed.
  • Fresh autoreview: clean, no accepted/actionable findings.

@vincentkoc
vincentkoc merged commit 8ff77c8 into openclaw:main Jun 10, 2026
181 of 186 checks passed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 11, 2026
…w#92027)

A chokidar watcher 'error' permanently disabled config hot-reload with a
single warn. Re-create the watcher with bounded backoff (500ms/2s/5s, 3
retries); on exhausted budget escalate to log.error and flip a persistent
hotReloadStatus to disabled. stop() clears any pending re-create timer.
badgerbees pushed a commit to badgerbees/openclaw that referenced this pull request Jul 8, 2026
…w#92027)

A chokidar watcher 'error' permanently disabled config hot-reload with a
single warn. Re-create the watcher with bounded backoff (500ms/2s/5s, 3
retries); on exhausted budget escalate to log.error and flip a persistent
hotReloadStatus to disabled. stop() clears any pending re-create timer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gateway Gateway runtime size: S triage: mock-only-proof Candidate: PR proof only shows tests, mocks, snapshots, lint, typecheck, or CI.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants