Skip to content

fix(agents): skip implicit provider discovery when models.mode is 'replace' [AI-assisted]#82638

Merged
vincentkoc merged 2 commits into
openclaw:mainfrom
eldar702:fix/66957-models-replace-skip-implicit
Jul 1, 2026
Merged

fix(agents): skip implicit provider discovery when models.mode is 'replace' [AI-assisted]#82638
vincentkoc merged 2 commits into
openclaw:mainfrom
eldar702:fix/66957-models-replace-skip-implicit

Conversation

@eldar702

Copy link
Copy Markdown
Contributor

Summary

  • Problem: src/agents/models-config.plan.ts:60-61 unconditionally awaits resolveImplicitProvidersImpl(...) (the provider-discovery scan) before reading cfg.models?.mode. When the user has set models.mode: "replace" to opt out of discovery, this still runs the full implicit-resolver — wasted work and (in some setups) 70+s of startup latency.
  • Why it matters: The documented contract at src/config/schema.help.ts:924 is that models.mode = "replace" uses only the configured providers. Honoring it eliminates the discovery cost for replace-mode users and fixes the timing regression they were filing the issue about.
  • What changed: Added a 3-line early-return at the top of resolveProvidersForModelsJsonWithDeps that short-circuits to mergeProviders({ implicit: {}, explicit: explicitProviders }) when cfg.models?.mode === "replace". The injected deps.resolveImplicitProviders seam used by tests is preserved for merge-mode callers.
  • What did NOT change (scope boundary): merge/undefined paths are untouched. planOpenClawModelsJsonWithDeps (which also reads cfg.models?.mode at L156) is unchanged — its existing logic short-circuits cleanly on the empty implicit result. No public type changes. No new exports.

Mirrors the sibling short-circuit at src/flows/model-picker.ts:116-118 (loadPickerModelCatalog does the same if (cfg.models?.mode === "replace") return ... before its catalog load).

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

(Closest match: agents config plumbing; no checkbox fits cleanly.)

Linked Issue/PR

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: resolveProvidersForModelsJsonWithDeps ran the implicit provider resolver even when models.mode === "replace", violating the documented contract and adding 70+s of startup latency in replace-mode setups.
  • Real environment tested: local clone built and tested on macOS arm64 (Node v24.7.0, pnpm 11.1.0, vitest 4.1.6).
  • Exact steps or command run after this patch:
    node scripts/run-vitest.mjs run \
      src/agents/models-config.replace-mode-skip-implicit-discovery.test.ts \
      src/agents/models-config.applies-config-env-vars.test.ts \
      src/agents/models-config.providers.implicit.discovery-scope.test.ts \
      src/agents/models-config.merge.test.ts
    
  • Evidence after fix:
     RUN  v4.1.6 /private/tmp/openclaw-triage-20pr/repo
    
     Test Files  2 passed (2)        # new test file (under 2 project lanes)
          Tests  6 passed (6)         # 3 scenarios × 2 lanes
     Test Files  6 passed (6)         # existing models-config tests
          Tests  46 passed (46)
    
    The new test file models-config.replace-mode-skip-implicit-discovery.test.ts has 3 scenarios:
    1. skips implicit discovery when models.mode === 'replace' — DI spy assertion not.toHaveBeenCalled, result contains only explicit providers.
    2. still resolves implicit when models.mode === 'merge' — spy called once, result merges both.
    3. still resolves implicit when models.mode is undefined (defaults to merge) — spy called once.
      Regression: all 46 existing tests across models-config.applies-config-env-vars.test.ts, models-config.providers.implicit.discovery-scope.test.ts, and models-config.merge.test.ts continue to pass — they use mode: undefined or merge, so they enter the same code path as before.
  • Observed result after fix: all 52 tests pass. With mode: "replace" and an injected spy, resolveImplicitProviders is never called.
  • What was not tested: I did not benchmark the wall-clock startup latency improvement in a real OpenClaw setup against the reporter's specific provider configuration (the issue body cites ~72s of pnpm tsx cold-load time). The new test proves the implicit-resolver call is genuinely skipped; converting that skip into measured-on-host latency win would require a configured agent with provider-discovery providers loaded.
  • Before evidence (optional but encouraged): on unpatched main, the new test 1 fails — not.toHaveBeenCalled is violated because resolveImplicitProvidersImpl is awaited unconditionally before the mode is read.

Root Cause (if applicable)

  • Root cause: the function reads explicitProviders first, then unconditionally awaits the implicit resolver, then merges. There is no mode check until planOpenClawModelsJsonWithDeps:156 — too late to skip the discovery work.
  • Missing detection / guardrail: documented contract at src/config/schema.help.ts:924 ("replace uses only your configured providers") was not enforced anywhere in resolveProvidersForModelsJsonWithDeps. The new test locks the contract via spy assertion.
  • Contributing context (if known): PR fix(agents): skip implicit provider discovery when models.mode is replace [AI-assisted] #73557 attempted a similar shape and was closed without merge. The shape used here mirrors the sibling pattern in src/flows/model-picker.ts:116-118, which is the canonical in-repo idiom for replace-mode short-circuits.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/agents/models-config.replace-mode-skip-implicit-discovery.test.ts (new).
  • Scenario the test should lock in: with cfg.models.mode === "replace", the injected resolveImplicitProviders spy must never be called and the result must contain only the explicit-provider entries.
  • Why this is the smallest reliable guardrail: uses the same DI seam (deps.resolveImplicitProviders) the codebase already uses for testing the merge path. No new mocking infrastructure required.
  • Existing test that already covers this (if any): none — models-config.applies-config-env-vars.test.ts and models-config.providers.implicit.discovery-scope.test.ts exercise the merge/undefined paths only.

User-visible / Behavior Changes

For users with models.mode: "replace":

  • Implicit provider discovery no longer runs. This is the documented contract.
  • Startup time drops by the cost of the discovery scan (reported as 70+s in some setups; varies by provider count and pnpm tsx cold-load behavior).
  • The final models.json contents are identical to what the merge-mode emitter would produce given zero implicit providers — i.e. the providers map is exactly the user's cfg.models.providers.

No change for merge/undefined mode users.

Diagram (if applicable)

N/A

AI Disclosure

This PR was authored with AI assistance (Claude Code, Opus 4.7). The fix shape was selected after a 5-lens design debate (MVD, defensive, pattern-match, performance, DX) — all five lenses converged on the same 3-line early-return mirroring src/flows/model-picker.ts:116-118. The chosen variant is the smallest viable diff that honors the documented contract.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S proof: supplied External PR includes structured after-fix real behavior proof. labels May 16, 2026
@clawsweeper

clawsweeper Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 1, 2026, 12:47 AM ET / 04:47 UTC.

Summary
The PR adds a replace-mode early return in agent model-provider planning and a focused Vitest file proving replace skips implicit discovery while merge/default still call it.

PR surface: Source +6, Tests +121. Total +127 across 2 files.

Reproducibility: yes. at source level. Current main still awaits implicit provider discovery in resolveProvidersForModelsJsonWithDeps before any replace-mode handling; this review did not rerun the reporter's wall-clock startup benchmark.

Review metrics: 1 noteworthy metric.

  • Config behavior surface: 1 existing mode behavior changed, 0 new options. models.mode replace now skips implicit provider discovery entirely, so maintainers should review upgrade impact even though no new config key is added.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #66957
Summary: This PR is the active fix candidate for the canonical replace-mode implicit provider discovery bug; related models-list work overlaps on replace-mode semantics but targets a distinct CLI display path.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Have a maintainer explicitly accept or reject the replace-mode compatibility impact before merge.

Risk before merge

  • [P1] Merging intentionally makes models.mode replace explicit-only during models.json planning; existing users who accidentally depended on runtime-discovered provider or plugin catalog rows under replace mode will stop receiving those rows after upgrade.

Maintainer options:

  1. Accept documented replace semantics (recommended)
    A maintainer can accept that replace mode now honors the documented explicit-only contract even if some existing setups lose unintended discovered provider rows.
  2. Request upgrade comparison first
    Before merging, ask for a before/after models.json comparison from a representative replace-mode setup that previously depended on discovered catalog rows.
  3. Pause for a compatibility mode
    If maintainers want to preserve the old discovered-row behavior, pause this PR and design an explicit opt-in compatibility path instead of merging the unconditional skip.

Next step before merge

  • No automated repair is needed; the remaining blocker is maintainer compatibility acceptance plus normal current-head merge validation.

Security
Cleared: The diff only changes a provider-planning branch and tests; it does not add dependencies, workflow execution, secret handling, or package-resolution changes.

Review details

Best possible solution:

Land the early replace-mode guard after maintainer compatibility acceptance, preserving merge/default discovery behavior and the focused regression coverage.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main still awaits implicit provider discovery in resolveProvidersForModelsJsonWithDeps before any replace-mode handling; this review did not rerun the reporter's wall-clock startup benchmark.

Is this the best way to solve the issue?

Yes, technically. The early guard is the narrowest maintainable fix because branching later in planOpenClawModelsJsonWithDeps is too late to avoid discovery; the remaining question is maintainer acceptance of the compatibility impact.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against d68ba5edc598.

Label changes

Label justifications:

  • P2: This fixes a documented provider-planning bug with large startup-delay impact, but the affected surface is limited to replace-mode model provider planning.
  • merge-risk: 🚨 compatibility: Existing replace-mode setups that relied on unintended implicit provider discovery can lose discovered provider/plugin catalog rows after upgrade.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body and follow-up comment include after-fix test output plus live production-resolver output showing replace mode skips implicit discovery while merge mode still runs it.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body and follow-up comment include after-fix test output plus live production-resolver output showing replace mode skips implicit discovery while merge mode still runs it.
Evidence reviewed

PR surface:

Source +6, Tests +121. Total +127 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 6 0 +6
Tests 1 121 0 +121
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 127 0 +127

What I checked:

  • Repository policy applied: Root AGENTS.md was read fully; its config/provider compatibility and whole-surface PR review guidance applies to this provider-planning change. (AGENTS.md:20, d68ba5edc598)
  • Scoped agents policy applied: The scoped agents guide treats broad provider/plugin runtime discovery in agent hot paths as performance-sensitive, directly matching this PR's discovery skip. (src/agents/AGENTS.md:1, d68ba5edc598)
  • Current-main bug path: On current main, resolveProvidersForModelsJsonWithDeps strips explicit providers, then selects and awaits resolveImplicitProvidersImpl before any replace-mode branch. (src/agents/models-config.plan.ts:115, d68ba5edc598)
  • Runtime caller path: prepareOpenClawModelsJsonSource calls planOpenClawModelsJson, so the unconditional discovery path is reached during models.json preparation for agents. (src/agents/models-config.ts:408, d68ba5edc598)
  • Implicit discovery work: resolveImplicitProviders resolves runtime plugin discovery providers and runs ordered discovery phases, which is the startup work replace mode should avoid. (src/agents/models-config.providers.implicit.ts:533, d68ba5edc598)
  • Documented config contract: Config help documents models.mode replace as using only configured providers, supporting the explicit-only behavior. (src/config/schema.help.ts:967, d68ba5edc598)

Likely related people:

  • steipete: Authored the provider-discovery helper split, lightweight provider discovery entries, and fast models-config env seam that define the affected planner and implicit resolver boundary. (role: feature-history contributor; confidence: high; commits: 7db79b04c673, fbbd644d7ae2, 53f6d962c25f; files: src/agents/models-config.plan.ts, src/agents/models-config.providers.implicit.ts, src/agents/models-config.applies-config-env-vars.test.ts)
  • chaoliang yan: Recently touched the same models.json planning surface for provider/baseUrl preservation behavior. (role: recent planner bugfix contributor; confidence: medium; commits: 35fb3f7e1c7b; files: src/agents/models-config.plan.ts)
  • Wynne668: Current checkout blame attributes the planner function to a recent commit that re-added the file in this shallow history, but the commit topic is adjacent rather than the feature origin. (role: recent area carrier; confidence: low; commits: ba3f68030b87; files: src/agents/models-config.plan.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added the P2 Normal backlog priority with limited blast radius. label May 16, 2026
@eldar702

eldar702 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Real-behavior proof — upgraded (production code path, no DI stubs)

Addressing the review note that the prior evidence used vi.fn() DI stubs only. This run drives the production resolveProvidersForModelsJsonWithDeps against the real resolveImplicitProviders (the actual discovery resolver) in both modes, on a clean checkout of this branch (macOS arm64, Node v24.7.0, tsx). The observer is transparent — it delegates verbatim to the production resolver and only records invocation count + wall-clock time. No canned return values.

Redacted runtime output:

=== models.mode = "replace" ===
  real resolveImplicitProviders invoked : 0 time(s)
  returned provider ids                 : [explicit]
  elapsed                               : 0.157 ms

[agents/model-providers] provider catalog timed out after 1500ms: codex; skipping provider discovery

=== models.mode = "merge" ===
  real resolveImplicitProviders invoked : 1 time(s)
  returned provider ids                 : [anthropic-vertex, explicit]
  elapsed                               : 47995.757 ms

=== VERDICT ===
  replace mode skips the real discovery resolver   : PASS
  merge mode still runs the real discovery resolver : PASS
  replace short-circuits faster than merge          : PASS

What this demonstrates with the real resolver (not a mock):

  • In replace mode the discovery resolver is invoked 0 times and the call returns in 0.157 ms with only the explicit provider.
  • In merge mode the real resolver runs (1 invocation), entering live provider discovery — here it discovered anthropic-vertex and hit a real catalog timeout for codex — taking ~48 s even with providerDiscoveryTimeoutMs: 1500.
  • That reproduces the 70+s startup latency from the linked issue on the unpatched (merge) path and shows the guard eliminates it for models.mode: "replace" users.

Honest scope: the ~48 s figure is this clean checkout's discovery cost and will vary with the reporter's configured providers/plugins. The behavioral contract the patch guarantees — replace mode performs zero implicit discovery — is what this run verifies against the production resolver. The colocated unit test also remains green (2 files / 6 tests passed).

(AI-assisted contribution; verified against current main before posting.)

@eldar702

eldar702 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jun 8, 2026
@eldar702
eldar702 marked this pull request as ready for review June 12, 2026 02:23
…place' [AI-assisted]

resolveProvidersForModelsJsonWithDeps unconditionally awaited the implicit
provider resolver before honoring models.mode. With mode: 'replace' the user
opts out of discovery, so add a 3-line early-return guard that returns only the
explicit providers — eliminating the (slow) discovery pass for replace-mode.

Closes openclaw#66957.
AI-assisted contribution.
@eldar702
eldar702 force-pushed the fix/66957-models-replace-skip-implicit branch from c2ab17f to 4472561 Compare June 12, 2026 02:25
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 12, 2026
@eldar702

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (the branch was far behind) — the fix is unchanged: a 3-line early-return in resolveProvidersForModelsJsonWithDeps that skips the implicit-provider discovery pass when models.mode === "replace". Verified on current main: the colocated test passes (3 passed), and the real-behavior proof posted above still holds (replace-mode → 0 discovery-resolver invocations / ~0.16ms vs ~48s in merge mode). Marking ready for review. @clawsweeper re-review

@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 12, 2026
@eldar702

Copy link
Copy Markdown
Contributor Author

Thanks for the clawsweeper re-review on June 14 — appreciated the detailed platinum-hermit verdict.

On the P1 risk clawsweeper flagged: existing replace-mode users who unintentionally relied on runtime-discovered provider/plugin catalog entries will stop receiving those entries after this patch — this is intentional and matches the documented replace-mode contract (explicit-only), but I want to surface it clearly for your decision.

If you want to confirm the behavior change is acceptable before merging, I am happy to provide a before/after models.json comparison from an existing replace-mode setup. Otherwise the patch is ready as-is. Let me know how you'd like to proceed.

@vincentkoc vincentkoc self-assigned this Jul 1, 2026
@vincentkoc
vincentkoc merged commit 5d52b5b into openclaw:main Jul 1, 2026
93 checks passed
@eldar702

eldar702 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the merge, @vincentkoc! 🙏 Glad the replace-mode skip-implicit-discovery fix is in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: models.mode="replace" still triggers implicit provider discovery and causes large startup delays

2 participants