Skip to content

fix(qa): stagger isolated worker startup#99294

Merged
RomneyDa merged 1 commit into
mainfrom
fix/qa-isolated-worker-stagger
Jul 3, 2026
Merged

fix(qa): stagger isolated worker startup#99294
RomneyDa merged 1 commit into
mainfrom
fix/qa-isolated-worker-stagger

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 3, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

The smoke-ci QA profile starts isolated flow workers only 500 ms apart. At concurrency 8, this clusters gateway startup and the delayed provider-auth worker prewarms, starving foreground gateway and channel work on the 4-vCPU CI runner.

The resulting failures present as unrelated timeouts, including skills.status, sessions.create, and native Telegram command processing. The gateway processes remain alive and are terminated only after the QA harness times out.

Why This Change Was Made

Restore the isolated-flow startup stagger to 1.5 seconds, matching the normal QA suite worker stagger. This spreads the initial eight gateway starts over 10.5 seconds instead of 3.5 seconds, reducing the startup/prewarm thundering herd without reducing scenario concurrency.

The 500 ms isolated override was introduced in 438f208a76 while optimizing unified QA suite runtime. Recent failures showed provider-auth prewarms taking 9–10.7 seconds with maximum event-loop stalls of 1.6–2.3 seconds.

User Impact

No product runtime behavior changes. QA smoke runs retain concurrency 8 and should avoid transient gateway/channel timeouts. The maximum scheduling delay for the initial eight-worker batch increases by approximately seven seconds and may be offset by reduced startup contention.

Evidence

  • Failure example: https://github.com/openclaw/openclaw/actions/runs/28630901850/job/84907411138
  • Additional affected runs showed sessions.create and native-command timeouts under the same 500 ms startup pattern.
  • pnpm test extensions/qa-lab/src/suite-planning.test.ts extensions/qa-lab/src/suite-launch.runtime.test.ts — 43 tests passed.
  • pnpm check — passed, including typecheck, lint, policy guards, and import-cycle checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant