Skip to content

fix(agents): classify provider upstream_error as fallbackable transient failure#95523

Closed
Bartok9 wants to merge 1 commit into
openclaw:mainfrom
Bartok9:fix/95519-upstream-error-fallback
Closed

fix(agents): classify provider upstream_error as fallbackable transient failure#95523
Bartok9 wants to merge 1 commit into
openclaw:mainfrom
Bartok9:fix/95519-upstream-error-fallback

Conversation

@Bartok9

@Bartok9 Bartok9 commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Classify provider upstream_error / "Upstream request failed" responses as a transient server failure so the configured fallback chain is attempted.
  • User-facing impact: a flaky primary provider that returns upstream_error now fails over to the next configured model instead of ending the turn with LLM request failed.

Motivation

Closes #95519.

When the primary provider returns an error payload shaped like:

{ "error": { "message": "Upstream request failed", "type": "upstream_error", "param": "", "code": null } }

the failover classifier returned null (unclassified). Unclassified failures do not enter the model-fallback path, so the run ended with stopReason: "error" and the user-facing text LLM request failed, with no model.fallback_step recorded — even when agents.defaults.model.fallbacks was configured.

The serverError matcher list already covered "upstream error" and "upstream connect error", but not the upstream_error type token nor the "Upstream request failed" message phrasing emitted by this provider. So this transient upstream failure slipped through.

The fix adds "upstream_error" and "upstream request failed" to the serverError patterns. These classify as a transient server_error (mapped to the timeout failover reason), which is the existing fallbackable transient path.

OpenClaw's own generic assistant text "LLM request failed" (GENERIC_ASSISTANT_ERROR_TEXT) is deliberately left unclassified to avoid a fallback loop on internal/non-provider failures.

Verification

  • node scripts/run-vitest.mjs run src/agents/embedded-agent-helpers/failover-matches.test.ts — 30 passed (5 new)
  • node scripts/run-vitest.mjs run src/agents/failover-error.test.ts src/agents/embedded-agent-helpers.isbillingerrormessage.test.ts — 234 passed
  • node scripts/run-vitest.mjs run src/agents/model-fallback.test.ts src/agents/embedded-agent-helpers/provider-error-patterns.test.ts — 121 passed
  • Did NOT change: the generic "LLM request failed" text classification (stays null).

Real behavior proof

  • Behavior addressed: the exact {type:"upstream_error", message:"Upstream request failed"} payload from the issue was classified null → no fallback. After the fix it classifies as timeout (fallbackable transient).
  • Real environment: Node v25.x, vitest 4.1.7, this branch.
  • Exact commands run after the patch:
    node scripts/run-vitest.mjs run <temp test calling classifyFailoverReason on the payloads>
    
  • Captured output AFTER fix:
    REASON payload = "timeout"
    REASON bare    = "timeout"
    REASON generic = null
    
    (payload = full issue JSON; bare = "Upstream request failed"; generic = OpenClaw's own "LLM request failed", correctly left unclassified.)
  • Before the fix, the same three inputs all returned null (verified on upstream/main before editing).
  • Regression test: src/agents/embedded-agent-helpers/failover-matches.test.tsdescribe("upstream provider transient errors (#95519)"). It asserts the new classifications AND that the generic text stays null; it fails without the source change.
  • What was NOT tested: end-to-end gateway fallback rotation (covered indirectly by the existing model-fallback.test.ts reason→candidate-rotation tests, which still pass); this PR is scoped to the classification gap that gates entry into that path.

What Problem This Solves

When the primary provider returns {type:"upstream_error", message:"Upstream request failed"}, the failover classifier returned null (unclassified), so the run ended with LLM request failed and never tried the configured fallback models — even with agents.defaults.model.fallbacks set. This PR classifies that payload as a transient server_error so the existing fallback path is entered.

Evidence

  • node scripts/run-vitest.mjs run src/agents/embedded-agent-helpers/failover-matches.test.ts30 passed (5 new, including the #95519 regression describe block). Re-verified on a clean rebase onto current upstream/main: 30/30 pass.
  • npx oxlint src/agents/embedded-agent-helpers/failover-matches.ts failover-matches.test.ts0 errors / 0 warnings.
  • Behavioral proof: classifyFailoverReason() on the exact issue payload returns "timeout" (fallbackable) AFTER the fix; returned null BEFORE. OpenClaw's own generic "LLM request failed" text correctly stays null (no fallback loop on internal errors).
  • Scope: classification gap only; downstream reason→candidate rotation remains covered by the existing passing model-fallback.test.ts.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 21, 2026
@clawsweeper

clawsweeper Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the branch is a useful matcher-only slice, but it does not cover the embedded provider-payload fallback boundary, while #95542 is an open, mergeable, proof-positive canonical fix for the same linked fallback bug.

Root-cause cluster
Relationship: superseded
Canonical: #95542
Summary: This PR is a narrower matcher-only candidate in the same upstream_error fallback cluster; the canonical open candidate covers the embedded runtime boundary and has positive proof.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Use #95542 as the canonical landing path for the linked fallback bug, then close the narrower matcher-only siblings as superseded.

So I’m closing this here and keeping the remaining discussion on #95542.

Review details

Best possible solution:

Use #95542 as the canonical landing path for the linked fallback bug, then close the narrower matcher-only siblings as superseded.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main routes embedded error payload text through an auth/auth_permanent/billing-only helper, and a null classification is treated as a completed model attempt. I did not run a live provider outage repro in this read-only review.

Is this the best way to solve the issue?

No: adding shared matcher strings is useful, but this branch is not the best fix because the linked runtime path also needs the embedded payload classifier to accept the structured upstream/server provider failure.

Security review:

Security review cleared: The diff only changes internal agent classifier source/tests and does not add dependencies, workflows, scripts, secrets handling, or code execution paths.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • vincentkoc: Local blame on the embedded payload classifier, server-error matcher region, and fallback-loop classification lines points to Vincent Koc in commit 91531ba; live history also shows adjacent model-fallback co-authorship. (role: current line-history contributor; confidence: medium; commits: 91531ba35c85, 9fd9aa5fcdd2, 7c97c6da9bfa; files: src/agents/embedded-agent-runner/result-fallback-classifier.ts, src/agents/embedded-agent-helpers/failover-matches.ts, src/agents/model-fallback.ts)
  • steipete: Live GitHub history shows recent work on the result-fallback classifier, model-fallback behavior, and model-failover docs that define this boundary. (role: recent fallback area contributor; confidence: high; commits: 0314819f918a, 6719528316c0, 4b0f16d496e5; files: src/agents/embedded-agent-runner/result-fallback-classifier.ts, src/agents/model-fallback.ts, docs/concepts/model-failover.md)
  • fuller-stack-dev: Recent commits co-authored or authored by this contributor touched structured provider error signal handling in the same error-helper surface. (role: structured provider-signal contributor; confidence: medium; commits: 89975eea24a5, 0314819f918a; files: src/agents/embedded-agent-helpers/errors.ts, src/agents/embedded-agent-helpers/errors-provider-structured-signals.test.ts)
  • takhoffman: The earlier embedded provider business-denial fallback classifier merge metadata names takhoffman as approved/co-authored, making them relevant to the same classifier boundary. (role: adjacent reviewer; confidence: medium; commits: 18f94fc83a72; files: src/agents/embedded-agent-runner/result-fallback-classifier.ts)

Codex review notes: model internal, reasoning high; reviewed against e6f41a4df005.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels Jun 21, 2026
@NianJiuZst

Copy link
Copy Markdown
Contributor

Complementary observations on top of ClawSweeper's review:

Same shape as #95521 — adds upstream_error / upstream_request_failed patterns to failover-matches.ts. The added test coverage (failover-matches.test.ts) is good.

…nt failure

When a provider returns an error payload shaped like
{"type":"upstream_error","message":"Upstream request failed"} the failover
classifier left it unclassified, so the turn ended with "LLM request failed"
and no configured fallback model was attempted.

The serverError pattern list already covered "upstream error" and
"upstream connect error" but not the "upstream_error" type token or the
"Upstream request failed" message phrasing, so these provider-side transient
failures fell through to the unclassified path that skips model fallback.

Add both patterns to serverError so they classify as a transient server_error
(mapped to the timeout failover reason), letting the fallback chain rotate to
the next candidate. OpenClaw's own generic "LLM request failed" text is
unaffected and stays unclassified to avoid fallback loops on internal failures.

Closes openclaw#95519
@Bartok9
Bartok9 force-pushed the fix/95519-upstream-error-fallback branch from 23f1231 to c57dccd Compare June 21, 2026 11:36
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 21, 2026
@clawsweeper

clawsweeper Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@Bartok9

Bartok9 commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Got it — thanks ClawSweeper. Agreed this is superseded; the fallback-classification intent is carried elsewhere. No objection to the close. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: XS status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Fallback should trigger on provider upstream_error / LLM request failed

2 participants