-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
exec auto-reviewer re-bills a fresh model completion for every byte-identical repeated exec #102249
Copy link
Copy link
Open
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.Issue affects a taxonomy feature currently scored M4/M5.
Type
Fields
Priority
None yet
Summary
When an agent has the model-backed exec auto-reviewer enabled, every ask-requiring exec that misses the allowlist and durable approval is sent to a fresh paid reviewer completion. The verdict is only ever registered as
allow-onceand is never memoized, so a loop that runs the same byte-identical command (for examplenpm test, or a poll/retry command) pays for one reviewer completion per invocation. N identical execs cost N paid reviewer calls.Environment
createModelExecAutoReviewerSteps to reproduce
autoReviewon and areviewer.modelso the model-backed exec reviewer is active.(command, argv, cwd, envKeys)tuple (for examplenpm teston two loop iterations).Expected
For an identical exec request within the same run, the deterministic (temperature 0) reviewer verdict should be reused, so the second identical exec costs zero additional reviewer completions.
Actual
The second identical exec re-enters the reviewer and fires a second paid completion. The first verdict is registered as
allow-once(node:registerNodeApproval+exec.approval.resolve; gateway:recordMatchedAllowlistUse), never as a durable approval and never as a per-run verdict memo, so nothing short-circuits the repeat.Root cause (if known)
src/agents/exec-auto-reviewer.tsruns a reviewer completion unconditionally per call, with no memo of the verdict keyed on the request tuple:Both call sites invoke this reviewer with no dedupe:
src/agents/bash-tools.exec-host-node.tscallsreviewer({ command, argv, cwd, envKeys, ... })and onallow-onceonly registers a single-use node approval.src/agents/bash-tools.exec-host-gateway.tscallsreviewer({ ... })and onallow-onceonly callsrecordMatchedAllowlistUse.Since the completion is
temperature: 0and the prompt is a pure function of the request tuple, memoizing the verdict for an identical(command, argv, cwd, envKeys)tuple for the lifetime of the run loses no security posture and removes the redundant paid calls.Sibling surfaces
Both exec hosts that dispatch the reviewer share the defect: node (
bash-tools.exec-host-node.ts) and gateway (bash-tools.exec-host-gateway.ts). The fix belongs in the shared reviewer (createModelExecAutoReviewer) so both hosts are covered by one memo rather than a per-caller cache.Scope / caveat
This only bites agents that have opted into the model-backed reviewer (
autoReviewplus a resolvablereviewer.model). Without a configured reviewer the path falls back todefaultExecAutoReviewerand no completion is billed.Real behavior proof
Behavior addressed: identical repeated exec requests re-bill a fresh reviewer completion instead of reusing the deterministic verdict.
Real environment tested: drove the real
createModelExecAutoReviewerreviewer and its real directive-check, dispatch, and parse logic at commit 534ace4, stubbing only the model client boundary (prepareSimpleCompletionModelForAgent/completeWithPreparedSimpleCompletionModel) to count completions and return a valid low-risk allow verdict.Exact steps or command run after this patch: built the production reviewer, invoked it twice with a byte-identical node-host input tuple (
command: "npm test",argv: ["npm","test"],cwd: "/work/project",envKeys: []), and wrote the observed decisions and completion count to out.txt. The EXPECTED half re-ran the identical scenario after adding a per-instance verdict memo keyed on the request tuple tocreateModelExecAutoReviewer, then reverted it.Evidence after fix:
Observed result after fix: two byte-identical execs bill two reviewer completions on current main; with a per-run verdict memo the second identical exec is served from the memo and bills one, with both verdicts unchanged.
What was not tested: I did not drive a full live provider completion (the model client boundary was counted, not called over the network), and did not measure a real multi-hundred-iteration agent loop; the defect is the per-call redundancy shown above scaling linearly with identical repeats.