Skip to content

test(qa): wait for durable webchat transcript#103718

Merged
steipete merged 1 commit into
mainfrom
codex/fix-qa-webchat-transcript-wait
Jul 10, 2026
Merged

test(qa): wait for durable webchat transcript#103718
steipete merged 1 commit into
mainfrom
codex/fix-qa-webchat-transcript-wait

Conversation

@steipete

Copy link
Copy Markdown
Contributor

What Problem This Solves

Closes #103701.

The live webchat runtime-parity scenario could pass after seeing transport output but before the assistant transcript row was durable. That left an assistant-required usage row at zero even though the scenario reported success.

Why This Change Was Made

The scenario now waits for the expected marker in the persisted session transcript. It retries only known not-yet-durable states and propagates deterministic read failures immediately. Behavior-level flow tests cover a missing file, an empty transcript, eventual persistence, and a non-retryable permission failure.

User Impact

Release QA no longer records false-success zero-usage webchat cells. The existing fail-closed runtime token gate and 15% threshold are unchanged.

Evidence

  • Local focused tests: 59 passed.
  • Testbox-through-Crabbox tbx_01kx63z1ngbqdygp3ewthcqsev: 59 focused tests plus all-lane changed checks, typecheck, lint, guards, and import-cycle checks passed.
  • Supporting authenticated runtime pair at ba1af773: OpenClaw and Codex both persisted WEBCHAT-DIRECT-REPLY-OK, with nonzero usage (14,683 and 21,663 tokens). Testbox run: https://github.com/openclaw/openclaw/actions/runs/29097879091
  • Fresh autoreview: no accepted or actionable findings.

@steipete

Copy link
Copy Markdown
Contributor Author

Land-ready proof for 5e5ba9dcebfbcc792a9f1afa64c71f09a8d27da9:

  • node scripts/run-vitest.mjs extensions/qa-lab/src/scenario-flow-runner.test.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/suite-runtime-agent-session.test.ts — 59 passed.
  • Testbox-through-Crabbox tbx_01kx63z1ngbqdygp3ewthcqsev — the same 59 focused tests plus all-lane check:changed, typecheck, lint, guards, and import-cycle checks passed.
  • Exact-head authenticated OpenClaw/Codex webchat-direct-reply-routing pair — 1/1 passed on Testbox tbx_01kx6581n6cd4exm4g93anj9k9; both transcripts persisted WEBCHAT-DIRECT-REPLY-OK and recorded nonzero usage (14,671 and 21,664 tokens): https://github.com/openclaw/openclaw/actions/runs/29098144719
  • Hosted CI — 43 passed, 12 skipped, zero failures: https://github.com/openclaw/openclaw/actions/runs/29098083918
  • Final branch autoreview — no accepted or actionable findings.

Known release gap: Codex token overhead still exceeds the unchanged 15% release threshold. This PR removes the false-success zero-usage race; it does not relax that gate or claim the broader token-efficiency blocker is resolved.

@steipete
steipete merged commit fbcb781 into main Jul 10, 2026
118 of 123 checks passed
@steipete
steipete deleted the codex/fix-qa-webchat-transcript-wait branch July 10, 2026 14:08
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

QA webchat parity can pass before assistant transcript usage is durable

1 participant