Skip to content

exec auto-reviewer re-bills a fresh model completion for every byte-identical repeated exec #102249

Description

@yetval

Summary

When an agent has the model-backed exec auto-reviewer enabled, every ask-requiring exec that misses the allowlist and durable approval is sent to a fresh paid reviewer completion. The verdict is only ever registered as allow-once and is never memoized, so a loop that runs the same byte-identical command (for example npm test, or a poll/retry command) pays for one reviewer completion per invocation. N identical execs cost N paid reviewer calls.

Environment

  • Commit: 534ace4 (origin/main at time of filing)
  • Surface: agent exec approval path (node and gateway hosts), createModelExecAutoReviewer

Steps to reproduce

  1. Configure an agent with autoReview on and a reviewer.model so the model-backed exec reviewer is active.
  2. Have the agent run the same non-allowlisted, ask-requiring command twice with an identical (command, argv, cwd, envKeys) tuple (for example npm test on two loop iterations).
  3. Observe that the reviewer issues a fresh paid completion on the second run even though the input is byte identical to the first.

Expected

For an identical exec request within the same run, the deterministic (temperature 0) reviewer verdict should be reused, so the second identical exec costs zero additional reviewer completions.

Actual

The second identical exec re-enters the reviewer and fires a second paid completion. The first verdict is registered as allow-once (node: registerNodeApproval + exec.approval.resolve; gateway: recordMatchedAllowlistUse), never as a durable approval and never as a per-run verdict memo, so nothing short-circuits the repeat.

Root cause (if known)

src/agents/exec-auto-reviewer.ts runs a reviewer completion unconditionally per call, with no memo of the verdict keyed on the request tuple:

completionController = new AbortController();
const result = await raceWithReviewerTimeout(
  complete({
    model: prepared.model,
    auth: prepared.auth,
    cfg,
    context: { systemPrompt: DEFAULT_EXEC_REVIEWER_SYSTEM_PROMPT, messages: [ /* buildReviewerUserPrompt(input) */ ] },
    options: { maxTokens: EXEC_REVIEWER_MAX_TOKENS, temperature: 0, signal: completionController.signal },
  }),
  { timeoutMs, onTimeout: () => completionController?.abort() },
);

Both call sites invoke this reviewer with no dedupe:

  • src/agents/bash-tools.exec-host-node.ts calls reviewer({ command, argv, cwd, envKeys, ... }) and on allow-once only registers a single-use node approval.
  • src/agents/bash-tools.exec-host-gateway.ts calls reviewer({ ... }) and on allow-once only calls recordMatchedAllowlistUse.

Since the completion is temperature: 0 and the prompt is a pure function of the request tuple, memoizing the verdict for an identical (command, argv, cwd, envKeys) tuple for the lifetime of the run loses no security posture and removes the redundant paid calls.

Sibling surfaces

Both exec hosts that dispatch the reviewer share the defect: node (bash-tools.exec-host-node.ts) and gateway (bash-tools.exec-host-gateway.ts). The fix belongs in the shared reviewer (createModelExecAutoReviewer) so both hosts are covered by one memo rather than a per-caller cache.

Scope / caveat

This only bites agents that have opted into the model-backed reviewer (autoReview plus a resolvable reviewer.model). Without a configured reviewer the path falls back to defaultExecAutoReviewer and no completion is billed.

Real behavior proof

Behavior addressed: identical repeated exec requests re-bill a fresh reviewer completion instead of reusing the deterministic verdict.
Real environment tested: drove the real createModelExecAutoReviewer reviewer and its real directive-check, dispatch, and parse logic at commit 534ace4, stubbing only the model client boundary (prepareSimpleCompletionModelForAgent / completeWithPreparedSimpleCompletionModel) to count completions and return a valid low-risk allow verdict.
Exact steps or command run after this patch: built the production reviewer, invoked it twice with a byte-identical node-host input tuple (command: "npm test", argv: ["npm","test"], cwd: "/work/project", envKeys: []), and wrote the observed decisions and completion count to out.txt. The EXPECTED half re-ran the identical scenario after adding a per-instance verdict memo keyed on the request tuple to createModelExecAutoReviewer, then reverted it.
Evidence after fix:

# OBSERVED (buggy, origin/main 534ace4d)
first.decision=allow-once risk=low
second.decision=allow-once risk=low
paid_reviewer_completions_for_2_identical_execs=2

# EXPECTED (identical inputs, verdict memoized per run)
first.decision=allow-once risk=low
second.decision=allow-once risk=low
paid_reviewer_completions_for_2_identical_execs=1

Observed result after fix: two byte-identical execs bill two reviewer completions on current main; with a per-run verdict memo the second identical exec is served from the memo and bills one, with both verdicts unchanged.
What was not tested: I did not drive a full live provider completion (the model client boundary was counted, not called over the network), and did not measure a real multi-hundred-iteration agent loop; the defect is the per-call redundancy shown above scaling linearly with identical repeats.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions