Skip to content

fix(coderd/x/chatd): show correct provider and clean detail for Bedrock errors#26338

Draft
johnstcn wants to merge 11 commits into
mainfrom
fix/bedrock-error-display
Draft

fix(coderd/x/chatd): show correct provider and clean detail for Bedrock errors#26338
johnstcn wants to merge 11 commits into
mainfrom
fix/bedrock-error-display

Conversation

@johnstcn

@johnstcn johnstcn commented Jun 12, 2026

Copy link
Copy Markdown
Member

Problem

Bedrock errors (routed through aibridge) showed the wrong provider ("Anthropic ...")
and a doubly-wrapped detail string instead of the clean message.

Fix (chatd only)

  • Provider label: thread the configured provider to error classification via
    GenerateAssistantOptions.ErrorProvider. Transport provider still drives prompt
    prep, sanitization, and metric labels (unchanged).
  • Detail: unwrap the SDK transport wrapper (METHOD "URL": NNN {body}) to surface
    the inner message; handles top-level {"message":...} and nested {"error":{"message":...}}.

Notes

  • Surfacing a top-level message now applies to all providers (intentional; nested wins when both present).
  • The advisor path keeps the transport label; accepted as-is (it returns err.Error(), not the classification).
  • Fixes display in chatd only; the wrapper originates in aibridge (out of scope here).

🤖 Generated by Coder Agents.

@johnstcn johnstcn self-assigned this Jun 12, 2026

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-06-15 17:55 UTC by @johnstcn
Spend: $80.85 / $100.00

Review history
  • R4 (2026-06-15), 3 Nit, 3 Note, 7 P3, 2 P4, COMMENT. Review
  • R5 (2026-06-15): 8 reviewers, 4 Nit, 3 Note, 8 P3, 2 P4, COMMENT. Review
  • R6 (2026-06-15): 8 reviewers, 4 Nit, 3 Note, 8 P3, 2 P4, APPROVE. Review

deep-review v0.7.1 | Round 6 | 2c75463..7a62f91

Last posted: Round 6, 17 findings (8 P3, 2 P4, 4 Nit, 3 Note), APPROVE. Review

Finding inventory

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Author fixed (c4d5c29) provider_error.go:80 IndexByte scans from string start, not after regex match; URL with { defeats unwrap R1 Mafuuu P3, Knov P3, Hisoka Note, Kite Note Yes
CRF-2 P3 Author fixed (2466009) chatloop.go:357 No test for ModelProvider override in GenerateAssistant R1 Bisky P3, Pariston P3 Yes
CRF-3 P3 Author fixed (86ec98c) chatloop.go:218 ModelProvider naming collision with ExecuteLocalToolsOptions.ModelProvider (different semantics) R1 Hisoka P3 Yes
CRF-4 P3 Author fixed (2466009) classify_test.go:1269 No test for transport wrapper fallback when JSON lacks message field R1 Chopper P3 Yes
CRF-5 P4 Author fixed (86ec98c) chatloop.go:352,generation.go:50,provider_error.go:17 Transport-vs-configured explanation repeated 3 times; trim to one canonical location R1 Gon P2 Yes
CRF-6 Nit Author fixed (86ec98c) chatloop.go:357 Use cmp.Or(opts.ModelProvider, provider) per codebase convention R1 Ging-Go Nit Yes
CRF-7 Nit Author fixed (c4d5c29) provider_error.go:17 Comment misattributes format to "Go HTTP client"; only Anthropic SDK produces it R1 Leorio Nit Yes
CRF-8 Nit Author fixed (c4d5c29) provider_error.go:90 jsonErrorMessage doc: "(AWS Bedrock)" on generic function; omits parse-order precedence R1 Gon P2, Leorio Nit Yes
CRF-9 Note Author accepted R3 (advisor path trade-off explicitly accepted) chatadvisor/runner.go:53 Advisor path doesn't thread ModelProvider; acknowledged in PR, no ticket R1 Chopper P3, Ryosuke P3, Knov P3, Bisky Note, Mafuuu Note, Pariston Note, Leorio Note, Kite Note Yes
CRF-10 Note Author fixed (89f441a) generation_preparer.go:590 debugProvider name carries user-visible semantics but keeps debug-oriented name R1 Mafu-san Note Yes
CRF-11 Note Author fixed (c4d5c29) provider_error.go:93 Duplicate anonymous struct for {error:{message:...}} in jsonErrorMessage and providerErrorResponseMessage R1 Robin Note Yes
CRF-12 P3 Author fixed (87a5aa8) chatloop_run_internal_test.go:174 Test name/comment still say ModelProvider after CRF-3 rename to ErrorProvider; points reader at the opposite-semantics field R3 Netero Yes
CRF-13 P3 Author fixed (87a5aa8) chatloop_run_internal_test.go:174 Override test proves error label flips but not that metric label stays on transport provider; retryable path unguarded R3 Bisky P3 Yes
CRF-14 P4 Author accepted R5 (plain field-copy seam; integration test disproportionate; overstated PR wording removed) generation.go:614 Relabel test stubs ErrorProvider directly; the resolveChatModel->prepareGeneration->generateAssistant seam is untested and can silently revert R3 Mafu-san P3, Meruem Note, Pariston Note Yes
CRF-15 P3 Author fixed (87a5aa8) provider_error.go:46 providerErrorDetail fallback returns raw providerErr.Message (the transport wrapper) without unwrapping; re-emits the bug when response-body extraction fails R3 Chopper P3 Yes
CRF-16 P3 Author fixed (3793be9) provider_error.go:94 jsonErrorMessage collapse to one unmarshal (CRF-11 fix, 7f86340) silently drops a usable top-level message when error is a non-object (e.g. {"error":"code","message":"..."}); re-emits the wrapper. Fix-chain regression R5 Chopper P3, Kite P3 Yes
CRF-17 Nit Author fixed (7a62f91) generation_preparer.go:462 CRF-10 rename left resolvedModel un-prefixed among cu-prefixed siblings (cuModel, cuDebugEnabled, cuResolvedProvider, cuErr) R5 Gon Nit Yes

Round log

Round 1

Panel. 0 Netero findings. 4 P3, 1 P4, 3 Nit, 3 Note. 2 Notes dropped (not posted). Reviewed against aab8c86..ed2e5f5.

Round 2

BLOCKED. 9 addressed, 2 silent (CRF-9, CRF-10). No review.

Round 3

PROCEED. CRF-10 addressed (89f441a), CRF-9 accepted by author. Panel re-review of round-2 fixes (first panel view) plus new test code.

Round 4

BLOCKED. CRF-12, CRF-13, CRF-15 addressed (87a5aa8). CRF-14 (P4) silent. No review.

Round 5

PROCEED. CRF-14 accepted by author. Panel verified round-3 fixes (CRF-12/13/15, first panel view) all hold. 2 new findings: CRF-16 (P3, fix-chain regression in the CRF-11 consolidation), CRF-17 (Nit).

Round 6

PROCEED. CRF-16 fixed (3793be9, json.RawMessage lazy decode + regression test), CRF-17 fixed (7a62f91, rename). Panel + Netero verified both fixes against current code with independent edge-case checks; no new findings. All 17 findings resolved or accepted. Approving.

Contested and acknowledged

CRF-9 (Note, chatadvisor/runner.go:53) - Advisor path does not thread ModelProvider

  • Finding: The advisor calls GenerateAssistant without setting the configured provider, so Bedrock errors through the advisor path fall back to Model.Provider() ("anthropic"). Review asked for a human decision.
  • Author accepted (R3): Author comment (IC_kwDOGkVX1s8AAAABGIcY4Q): "CRF-9: accepting the trade-off here." Advisor errors surface as a raw err.Error() string in AdvisorResult.Error, not the structured ClassifiedError, so the mislabeled provider is not directly user-visible. Trade-off explicitly accepted; no ticket.

CRF-14 (P4, generation.go:614) - Untested resolve->prepare->generate seam

  • Finding: The relabel test stubs ErrorProvider: "bedrock" directly; no test covers that prepareGeneration populates ResolvedProvider and generateAssistant forwards it. Deleting the field copy silently reverts the relabel. The PR description overclaimed "end to end" coverage.
  • Author accepted (R5): Thread reply PRRC_kwDOGkVX1s7LkjhK and comment IC_kwDOGkVX1s8AAAABGMXAbw: accepts the gap as a plain field-copy seam where a full taskStarter integration test would be disproportionate (per the panel's own note), with behavior covered at the chatloop layer plus parsing in chaterror, and removed the overstated "end to end" wording from the PR description.
About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean fix, well-scoped. The transport-vs-configured provider split is the right architectural move, and Fix B's defensive parsing degrades gracefully if aibridge is fixed upstream. Tests cover both the positive path and the guard against false matches. CI is green.

Severity summary: 4 P3, 1 P4, 3 Nit, 3 Note.

The chatadvisor path (runner.go:53) still constructs GenerateAssistantOptions without ModelProvider, so Bedrock errors through the advisor would still say "Anthropic." The PR description acknowledges this. Eight reviewers flagged it (three at P3, five at Note). The advisor error surfaces as a raw string, not the structured classification, limiting the user-visible impact. This needs a human decision: file a ticket or explicitly accept the gap.

"If the URL portion contains { (e.g., a templated URL like https://example.com/{resource}/invoke), IndexByte finds that { instead of the JSON body's opening brace. The substring from that position is not valid JSON, so json.Unmarshal fails and the function falls back to returning the original ugly wrapper." (Mafuuu)


coderd/x/chatd/chatadvisor/runner.go:53

Note [CRF-9] The advisor constructs GenerateAssistantOptions without ModelProvider, so Bedrock errors through the advisor path still display "Anthropic." The PR description acknowledges this. The advisor error surfaces as err.Error() into AdvisorResult.Error, not as the structured ClassifiedError payload, which limits user-visible impact. This needs a human decision: file a ticket to track or explicitly accept the gap.

(Chopper P3, Ryosuke P3, Knov P3, Bisky Note, Mafuuu Note, Pariston Note, Leorio Note, Kite Note)

🤖

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chaterror/provider_error.go Outdated
Comment thread coderd/x/chatd/chatloop/chatloop.go Outdated
Comment thread coderd/x/chatd/chatloop/chatloop.go Outdated
Comment thread coderd/x/chatd/chaterror/classify_test.go
Comment thread coderd/x/chatd/chatloop/chatloop.go Outdated
Comment thread coderd/x/chatd/chatloop/chatloop.go Outdated
Comment thread coderd/x/chatd/chaterror/provider_error.go Outdated
Comment thread coderd/x/chatd/chaterror/provider_error.go Outdated
Comment thread coderd/x/chatd/generation_preparer.go Outdated
Comment thread coderd/x/chatd/chaterror/provider_error.go Outdated

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good progress. 9 of 11 findings from round 1 are addressed (all P3, P4, Nits, and one Note fixed). Two Notes remain without author response:

CRF-9 (Note, chatadvisor/runner.go:53): Advisor path doesn't thread ModelProvider. The round 1 review asked for a human decision: file a ticket or explicitly accept the gap. No response yet.

CRF-10 (Note, generation_preparer.go:590): debugProvider variable name carries user-visible semantics but keeps its debug-oriented name. No response yet.

Further review is blocked until the author responds to these two findings. A brief acknowledgment ("accepted as-is" or "will file a ticket") is sufficient to unblock.

🤖 This review was automatically generated with Coder Agents.

@johnstcn

Copy link
Copy Markdown
Member Author

CRF-9: accepting the trade-off here.

@johnstcn

Copy link
Copy Markdown
Member Author

/coder-agents-review model:claude-opus-4-8 thinking:high

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strong round. Every round-1 finding the panel could verify is genuinely fixed at the root, not patched over: CRF-1's brace search is now anchored after the regex match with a dedicated regression test, the CRF-3 field rename eliminates the ErrorProvider/ModelProvider collision, CRF-5/7/8 comment fixes landed cleanly, CRF-10's debugProvider rename propagated fully (including the computer-use local), and CRF-11's duplicate struct is consolidated. CRF-9 was accepted by the author with a stated trade-off, and CRF-10 was fixed in 89f441a. Tests pass and the new chaterror suite is well-shaped.

This is the panel's first look at the round-2 fix commits (round 2 was blocked on author response), and it surfaced four new findings, all on the new test and fallback code: 3 P3, 1 P4.

Two themes worth calling out. First, the CRF-2 override test proves only half of Fix A: it asserts the error label flips to "bedrock" but never that the metric label stays on the transport provider, the documented invariant ("Metric labels intentionally still use the transport provider"). The retryable path where the two must diverge is exactly the case the test avoids. Second, the PR description says the relabel test "covers the provider relabel wiring end to end," but the test stubs ErrorProvider: "bedrock" directly and never exercises the resolveChatModel -> prepareGeneration -> generateAssistant resolution path; that claim overstates the coverage.

"swap provider for errorProvider in both RecordStreamRetry calls to 'simplify,' and every test stays green while Bedrock retries silently split off into a new 'bedrock' metric series and break the dashboards the PR set out to preserve." (Bisky)

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chaterror/provider_error.go Outdated
Comment thread coderd/x/chatd/chatloop/chatloop_run_internal_test.go Outdated
Comment thread coderd/x/chatd/chatloop/chatloop_run_internal_test.go Outdated
Comment thread coderd/x/chatd/generation.go

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three round-3 P3 findings are resolved in 87a5aa8: CRF-12 (test renamed to ..._ErrorProviderOverridesTransportLabel, comment corrected), CRF-13 (TestGenerateAssistant_StreamRetryRecordsMetric now sets ErrorProvider: "bedrock" and asserts both that the classified error uses "bedrock" and that the retry metric keeps provider: "test-provider"), and CRF-15 (the providerErrorDetail fallback now calls unwrapTransportErrorMessage, with TestClassify_UnwrapsTransportWrapperInMessageFallback locking it in). Nice, targeted fixes.

One finding is unaddressed and has no response:

CRF-14 (P4, generation.go:614): the resolveChatModel -> prepareGeneration -> generateAssistant seam that populates ResolvedProvider/ErrorProvider is untested, so deleting the field copy would silently revert the relabel. The reviewers explicitly said a full taskStarter integration test would be disproportionate, so this does not require code. It needs only a one-line decision: accept the gap as-is, or file a ticket. A bare acknowledgment unblocks the panel.

Further review is blocked until CRF-14 gets a response (accept the gap, file a ticket, or push a test).

🤖 This review was automatically generated with Coder Agents.

johnstcn commented Jun 15, 2026

Copy link
Copy Markdown
Member Author

CRF-14: accepting the gap as-is. The prepareGeneration -> ErrorProvider seam is a plain field copy; per the panel's own note a full taskStarter integration test would be disproportionate. The behavior is covered at the chatloop layer (TestGenerateAssistant_ErrorProviderOverridesTransportLabel, plus the metric-divergence guard) and parsing in chaterror. The PR description's overstated "end to end" wording has been removed.

Posted by Coder Agents on behalf of @johnstcn.

(Edit by the human in question: I approve of this message)

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First panel view of the round-3 fix commits (rounds 2 and 4 were churn-guard blocked). All three round-3 P3 fixes verified by the panel, with mutation checks: CRF-12 (test renamed and comment corrected), CRF-13 (the metric-vs-error invariant is now genuinely guarded; the retryable path fires and asserts Classify(err).Provider == "bedrock" while the retry counter keeps provider: "test-provider", so a swap of either label turns the test red), and CRF-15 (the providerErrorDetail message fallback now unwraps the transport wrapper, with a non-tautological test). The accepted gaps CRF-9 and CRF-14 still hold against the current code. Nice, root-cause fixes.

One regression surfaced, introduced by a later commit. 1 P3, 1 Nit.

Commit 7f86340, labeled refactor ("collapse jsonErrorMessage to one unmarshal"), is not behavior-preserving. It addressed CRF-11 (a Note about a duplicate struct) but narrowed the function's tolerance: a single decode into the combined struct fails whenever error is present as a non-object (string, array, number), silently dropping a usable top-level message. Two reviewers caught this independently with side-by-side old-vs-new output, and I reproduced it: {"message":"rate limited","error":"rate_limit"} returns the message under the old two-pass code and "" under the new code. The fix-chain went net-negative here: a Note-level cleanup introduced a P3 that re-emits the exact bug this PR fixes.

"a 'refactor' commit should not introduce a silent behavioral divergence. The existing tests don't cover a non-object error alongside a message, so the regression is invisible." (Kite)

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/x/chatd/chaterror/provider_error.go
Comment thread coderd/x/chatd/generation_preparer.go Outdated

Copy link
Copy Markdown
Member Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean verification round, and the end of the line for this PR's findings. Both round-5 fixes are verified complete by the panel and Netero, each against the current code rather than the author's word:

CRF-16 (P3, the fix-chain regression): jsonErrorMessage now decodes error as json.RawMessage and attempts the nested decode lazily, falling back to the top-level message when error is a non-object. This restores the two-pass tolerance the CRF-11 consolidation dropped without reintroducing the duplicate struct. Five reviewers independently traced the branch table (non-object string/array/number/null error, empty {}, nested precedence) and confirmed TestClassify_KeepsTopLevelMessageWhenErrorIsNonObject fails under the old code and passes under the new. Nested precedence stays guarded.

CRF-17 (Nit): resolvedModel renamed to cuResolvedModel; the computer-use destructuring is now uniformly cu-prefixed.

The two author-accepted gaps (CRF-9 advisor path, CRF-14 field-copy seam) still hold against the current code; no reviewer found new evidence to reopen them.

Across six rounds this PR took 17 findings (1 of them a fix-chain regression introduced by an earlier fix) to root-cause resolution or explicit human acceptance, each fix shipping with a regression test. All findings are now resolved or accepted. Nothing left on the board.

"Boring, in the way a closed case is boring." (Hisoka)

🤖 This review was automatically generated with Coder Agents.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant