Skip to content

Surface config hot-reload watcher status in health#99267

Merged
vincentkoc merged 4 commits into
openclaw:mainfrom
masatohoshino:fix/config-hot-reload-health-status
Jul 6, 2026
Merged

Surface config hot-reload watcher status in health#99267
vincentkoc merged 4 commits into
openclaw:mainfrom
masatohoshino:fix/config-hot-reload-health-status

Conversation

@masatohoshino

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes an issue where operators running a managed gateway would keep working
with a silently disabled config hot-reload watcher. #92027 and #92852
(both merged) taught the watcher to retry past transient native/polling
errors and record a terminal "disabled" status once retries are
exhausted, but that status was discarded at the managed-reloader boundary
and never reached openclaw health. An operator editing openclaw.json
and expecting a hot reload would see nothing happen, with no way to tell
"the gateway ignored this edit" apart from "the edit produced a no-op
reload plan."

Why This Change Was Made

The watcher already computes the right terminal state; this change is pure
plumbing to surface it, not new reload policy.

Changed surface:

  • src/gateway/server-runtime-handles.ts / src/gateway/server-reload-handlers.ts
    GatewayConfigReloaderHandle gains an optional hotReloadStatus()
    accessor alongside stop(); startManagedGatewayConfigReloader returns
    the watcher's real accessor instead of a handle that only exposes stop.
  • src/commands/health.types.ts / src/commands/health.ts — new optional
    HealthSummary.configReload.hotReloadStatus field + a text-output line
    that only prints on "disabled".
  • src/gateway/server.impl.ts / src/gateway/server/health-state.ts /
    src/gateway/server-methods/shared-types.ts /
    src/gateway/server-request-context.ts / src/gateway/server-methods/health.ts
    — thread the live accessor through both the full-refresh path and the
    cache-hit merge path (one new GatewayRequestContext getter, wired at the
    single createGatewayRequestContext call site).

Each layer above exists because the health RPC has two independent response
paths (full refresh vs. cache-hit merge), and both already carry other live
runtime facts (eventLoop, modelPricing, contextEngines) the same way —
this change reuses that existing plumbing pattern rather than adding a new
one. No generic "conditions" abstraction was introduced because there is
exactly one signal to surface today; a future second signal would be the
trigger to generalize.

configReload.hotReloadStatus follows the existing modelPricing/
contextEngines sibling pattern: the JSON field is present whenever a real
reloader is running ("active" or "disabled"), and omitted only when
there is no reloader at all (minimalTestGateway, pre-startup inert state).
Text output additionally suppresses the "active" case and only prints a
line on "disabled", matching how formatModelPricingHealthLine and
formatContextEngineHealthLine stay silent on the healthy/ok case.

Non-goals: no new persisted state (the field is derived, not stored), no
doctor repair (this surfaces existing in-memory watcher state, not a
migratable config shape), no generic "conditions" enum (see above), no
change to reload/retry policy.

User Impact

Operators can now run openclaw health (text or --json) and see Config hot reload: disabled (watcher retries exhausted; restart the gateway to restore it) when the gateway's config watcher has permanently given up,
instead of silently continuing to run on stale config with no visible
signal. The status is fresh on every call, including cache-hit responses,
so it's visible immediately rather than after a cache refresh delay.
--json includes configReload.hotReloadStatus whenever a reloader is
running, whether "active" or "disabled"; text output only prints a line
for the "disabled" case, so the common healthy case stays quiet. The
field is omitted entirely only when no reloader exists at all (e.g.
minimal/test gateways).

Evidence

Production-path health capture:

  • Booted a real gateway on this branch (isolated OPENCLAW_STATE_DIR/
    OPENCLAW_CONFIG_PATH/port, gateway.auth.mode=none, channels/providers
    skipped so no real credentials were needed) and ran openclaw health --json; the response included "configReload":{"hotReloadStatus":"active"}.
    This validates the "active" plumbing end to end through the real
    production chain (real startGatewayConfigReloader
    runtimeState.configReloaderGatewayRequestContext
    healthHandlers.health → JSON output), not just a mocked/unit-test path.
  • This capture does not by itself validate the "disabled" path in a
    running production gateway — that terminal transition is covered
    separately by src/gateway/config-reload.test.ts's existing real-watcher
    retry/polling exhaustion test (cited below), not a live capture. Forcing
    real chokidar/inotify exhaustion in a production-path capture would be
    flaky and wasn't attempted.

Existing signal proof (unchanged, cited not duplicated):
src/gateway/config-reload.test.ts:1675"degrades to polling then disables after both native and polling retries are exhausted" — already
proves hotReloadStatus() flips "active""disabled" after native +
polling retries are exhausted (introduced by #92027/#92852).

New plumbing proof (src/gateway/server-reload-handlers.hot-reload-status.test.ts):
mocks startGatewayConfigReloader to return a controllable live accessor,
then proves startManagedGatewayConfigReloader(...).hotReloadStatus is a
function that reflects a live mutation of the underlying watcher's status
("active""disabled") without recreating the managed handle — this is
exactly the regression a copied/snapshotted value would fail.

New health summary proof (src/commands/health.snapshot.test.ts):

  • "omits configReload when no config reloader status is supplied"
  • "surfaces a disabled config hot-reload watcher in the health snapshot"

New health-state passthrough proof (src/gateway/server/health-state.test.ts):

  • "passes the config reloader hot-reload status only when the hook returns one"
    (mirrors the existing getEventLoopHealth passthrough test)

New cache-hit freshness proof (src/gateway/server-methods/server-methods.test.ts,
describe("gateway healthHandlers.health cache freshness")):

  • "merges a live disabled config hot-reload status into cached health responses"
    — cached snapshot says "active", live accessor says "disabled"; the
    cache-hit response ({ cached: true }) reflects "disabled" immediately
  • "preserves the cached config hot-reload status when no live accessor is available" — no getter on context → last cached value survives unchanged

New live request-context read proof (src/gateway/server-request-context.test.ts):

  • "reads config hot-reload status live from runtime state" — proves
    context.getConfigReloaderHotReloadStatus() reflects live mutation of
    runtimeState.configReloader at the single wiring site, mirroring the
    existing live cron / cronStorePath request-context read tests.

New CLI output proof (src/commands/health.test.ts):

  • formatConfigReloadHealthLine unit tests: disabled → line text, active →
    null, absent → null
  • "surfaces a disabled config hot-reload watcher in JSON output"
  • "prints the config hot-reload disabled line in text output"
  • "omits the config hot-reload line in text output when the reloader is active"

Validation run (worktree repo/worktrees/config-hot-reload-health-status,
node scripts/run-vitest.mjs, rebased onto upstream/main
dad7768da8500d1c785ef33e19aabee9d01081bd):

  • src/gateway/config-reload.test.ts — 304/304 passed
  • src/gateway/server-reload-handlers.test.ts +
    src/gateway/server-reload-handlers.hot-reload-status.test.ts +
    src/gateway/server-request-context.test.ts +
    src/gateway/server-methods/server-methods.test.ts +
    src/gateway/server/health-state.test.ts (gateway config project,
    15 files) — 553/553 passed
  • src/commands/health.test.ts + src/commands/health.snapshot.test.ts
    35/35 passed
  • pnpm exec oxfmt --check on all touched files — clean
  • node scripts/run-oxlint.mjs on all touched files — clean
  • node scripts/run-tsgo.mjs -p tsconfig.core.json (prod typecheck) — clean
  • node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json (test
    typecheck) — clean
  • node --import tsx scripts/check-import-cycles.ts — 0 runtime value cycles

No live/E2E proof was added beyond the production-path capture above: this
is plumbing of an already-tested terminal state through existing accessor
boundaries, not new watcher/chokidar behavior, so no OS-specific or
timing-sensitive CI test was introduced per repo guidance against flaky
watcher tests in CI.

Precheck note

  • ocw review precheck config-hot-reload-health-status reached the broad
    gateway-server Vitest lane but failed in pre-existing shared-lane tests
    unrelated to this branch.
  • Minimal reproducer with zero files from this branch:
    node scripts/run-vitest.mjs run --config test/vitest/vitest.gateway-server.config.ts src/gateway/server.health.test.ts src/gateway/server-startup-log.test.ts
    reproduces the same server-startup-log.test.ts warning failures.
  • The reproducer files and root-cause helper are byte-identical to
    upstream/main; this branch does not touch them.
  • Root cause: src/gateway/test-helpers.mocks.ts mutates
    process.env.OPENCLAW_SKIP_CHANNELS = "1" at module top level without
    cleanup, then leaks across the non-isolated gateway-server project
    depending on file order.
  • This branch's own focused tests, typecheck, lint, import-cycle checks,
    and production-path health capture all passed.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime commands Command implementations size: M labels Jul 3, 2026
@clawsweeper

clawsweeper Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 2, 2026, 9:17 PM ET / 01:17 UTC.

Summary
The branch threads the existing config hot-reload watcher status into managed gateway handles, health snapshots, cached health RPC responses, CLI text/JSON output, and focused tests.

PR surface: Source +51, Tests +325. Total +376 across 17 files.

Reproducibility: yes. Source inspection shows current main tracks hotReloadStatus() in the config watcher but drops it at the managed reloader and health paths; I did not run a local gateway because this review is read-only.

Review metrics: 1 noteworthy metric.

  • Health payload surface: 1 optional JSON field added. openclaw health --json is operator-facing and may be consumed by scripts, so maintainers should notice the additive payload shape before merge.

Stored data model
Persistent data-model change detected: serialized state: src/commands/health.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #99267
Summary: This PR is the canonical active item for surfacing the existing config watcher terminal status in health; the related merged PRs are prerequisites that added the underlying watcher behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Next step before merge

  • No automated repair lane is needed; the current head has no blocking findings and is ready for normal maintainer review/merge judgment.

Security
Cleared: The diff adds typed in-process health plumbing and tests without changing secrets, auth, dependencies, CI, install scripts, package resolution, or downloaded code.

Review details

Best possible solution:

Land the narrow additive health plumbing after normal maintainer approval, keeping the signal derived from live runtime state and the status type in a leaf contract.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main tracks hotReloadStatus() in the config watcher but drops it at the managed reloader and health paths; I did not run a local gateway because this review is read-only.

Is this the best way to solve the issue?

Yes. Forwarding the existing live accessor through the managed handle, full-refresh path, and cached health merge path is the narrow owner-boundary fix, and the leaf type avoids the earlier implementation-cycle problem.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 381b44a9fc26.

Label changes

Label changes:

  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix live gateway health JSON output for the active status, with focused tests covering disabled, cached, JSON, and text-output paths.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P2: This is a normal-priority gateway health diagnostic bug fix with limited blast radius and current CI/proof support.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix live gateway health JSON output for the active status, with focused tests covering disabled, cached, JSON, and text-output paths.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live gateway health JSON output for the active status, with focused tests covering disabled, cached, JSON, and text-output paths.
Evidence reviewed

PR surface:

Source +51, Tests +325. Total +376 across 17 files.

View PR surface stats
Area Files Added Removed Net
Source 11 59 8 +51
Tests 6 327 2 +325
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 17 386 10 +376

What I checked:

  • Root and scoped policy read: Read the full root AGENTS.md plus the scoped Gateway and server-methods AGENTS.md files; their gateway review and hot-path coupling guidance applies to this PR. (AGENTS.md:1, 381b44a9fc26)
  • Current main has the underlying status but drops it: Current main's config reloader exposes a live hotReloadStatus accessor, while startManagedGatewayConfigReloader returns only stop(), so health cannot observe the disabled watcher state yet. (src/gateway/server-reload-handlers.ts:835, 381b44a9fc26)
  • PR forwards the live managed reloader accessor: At PR head, the managed reloader handle includes hotReloadStatus: configReloader.hotReloadStatus while still preserving the existing stop cleanup path. (src/gateway/server-reload-handlers.ts:844, 3325910a42b8)
  • PR surfaces fresh status in both health paths: PR head passes the live status into full health refreshes and merges it into cached health responses, matching the existing eventLoop-style live-runtime pattern. (src/gateway/server-methods/health.ts:117, 3325910a42b8)
  • Leaf status type resolves the earlier architecture issue: The new config-reload-status.types.ts is a leaf contract for the union type, so shared health/server-method types no longer import the config-reload implementation module. (src/gateway/config-reload-status.types.ts:1, 3325910a42b8)
  • Live CI/proof check: Live PR checks for the current head show the previous check-additional-runtime-topology-architecture blocker passing, along with prod types, test types, lint, and the real behavior proof gate. (3325910a42b8)

Likely related people:

  • hansraj316: Authored the merged watcher recovery PR that added the persistent hotReloadStatus signal this PR now surfaces. (role: introduced behavior; confidence: high; commits: 8ff77c8168e4; files: src/gateway/config-reload.ts, src/gateway/config-reload.test.ts)
  • danbao: Authored the merged polling fallback work that defines the native-to-polling-to-disabled watcher path reported by this health signal. (role: adjacent feature contributor; confidence: high; commits: c78f9376d929; files: src/gateway/config-reload.ts, src/gateway/config-reload.test.ts)
  • vincentkoc: Merged the prerequisite watcher PRs and has nearby config-reload history including effective polling-mode tracking. (role: reviewer, merger, and adjacent contributor; confidence: high; commits: 8ff77c8168e4, c78f9376d929, 40093f5a93ed; files: src/gateway/config-reload.ts, src/gateway/config-reload.test.ts)
  • steipete: Recent GitHub path history shows repeated work around gateway health, server-method boundaries, and gateway protocol extraction near this owner boundary. (role: recent area contributor; confidence: medium; commits: a84910be9149, f24ae91842d6, b1117d98622f; files: src/gateway/server-methods/health.ts, src/gateway/server/health-state.ts, src/commands/health.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 3, 2026
@vincentkoc vincentkoc self-assigned this Jul 6, 2026
Thread the config reloader's existing hotReloadStatus() accessor through
the managed reloader handle (previously only stop() survived the handoff)
and into openclaw health as an optional configReload.hotReloadStatus
field, so operators can tell when the watcher has permanently disabled
itself after exhausting retries instead of silently running with stale
config.
…h responses

Thread getConfigReloaderHotReloadStatus through GatewayRequestContext
(mirroring the existing getEventLoopHealth pattern) so the health RPC's
cache-hit merge path picks up a live watcher disable within the same
request instead of waiting up to HEALTH_REFRESH_INTERVAL_MS for the
background refresh to catch up.
Move GatewayHotReloadStatus out of config-reload.ts into a leaf
config-reload-status.types.ts so health/request-context/runtime-handles
callers no longer pull the full config-reload implementation into the
shared server-method type graph (ClawSweeper-flagged architecture cycle).
@vincentkoc
vincentkoc force-pushed the fix/config-hot-reload-health-status branch from 3325910 to cd4027a Compare July 6, 2026 08:35
@vincentkoc

Copy link
Copy Markdown
Member

Maintainer update: rebased the contributor branch onto current main, preserved both the new delivery-queue health surface and this PR's config-reload health surface, then updated the new hot-reload test fixture for the latest ManagedGatewayConfigReloaderParams contract. Exact head f21de47483e652cff3427953b38bd067e660500d passes 245 focused tests across gateway/commands, the focused 2-test hot-reload suite, core test typechecking, oxfmt, oxlint, and git diff --check. No CI failures remain; exact-head auto-merge is being armed for the remaining protection checks.

@vincentkoc
vincentkoc merged commit e7158a5 into openclaw:main Jul 6, 2026
95 checks passed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 7, 2026
* fix(gateway): surface config hot-reload watcher status in health

Thread the config reloader's existing hotReloadStatus() accessor through
the managed reloader handle (previously only stop() survived the handoff)
and into openclaw health as an optional configReload.hotReloadStatus
field, so operators can tell when the watcher has permanently disabled
itself after exhausting retries instead of silently running with stale
config.

* fix(gateway): keep configReload.hotReloadStatus fresh on cached health responses

Thread getConfigReloaderHotReloadStatus through GatewayRequestContext
(mirroring the existing getEventLoopHealth pattern) so the health RPC's
cache-hit merge path picks up a live watcher disable within the same
request instead of waiting up to HEALTH_REFRESH_INTERVAL_MS for the
background refresh to catch up.

* fix(health): move config reload status type to leaf contract

Move GatewayHotReloadStatus out of config-reload.ts into a leaf
config-reload-status.types.ts so health/request-context/runtime-handles
callers no longer pull the full config-reload implementation into the
shared server-method type graph (ClawSweeper-flagged architecture cycle).

* test(gateway): update config reloader fixture

---------

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
giodl73-repo pushed a commit to giodl73-repo/openclaw that referenced this pull request Jul 8, 2026
* fix(gateway): surface config hot-reload watcher status in health

Thread the config reloader's existing hotReloadStatus() accessor through
the managed reloader handle (previously only stop() survived the handoff)
and into openclaw health as an optional configReload.hotReloadStatus
field, so operators can tell when the watcher has permanently disabled
itself after exhausting retries instead of silently running with stale
config.

* fix(gateway): keep configReload.hotReloadStatus fresh on cached health responses

Thread getConfigReloaderHotReloadStatus through GatewayRequestContext
(mirroring the existing getEventLoopHealth pattern) so the health RPC's
cache-hit merge path picks up a live watcher disable within the same
request instead of waiting up to HEALTH_REFRESH_INTERVAL_MS for the
background refresh to catch up.

* fix(health): move config reload status type to leaf contract

Move GatewayHotReloadStatus out of config-reload.ts into a leaf
config-reload-status.types.ts so health/request-context/runtime-handles
callers no longer pull the full config-reload implementation into the
shared server-method type graph (ClawSweeper-flagged architecture cycle).

* test(gateway): update config reloader fixture

---------

Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands Command implementations gateway Gateway runtime P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants