chore(evals): fix stale docs and drifted Makefile defaults#1237
Merged
Conversation
The local mcpchecker eval docs and Makefile defaults drifted after the ACP migration (containers#839) and the kubernetes -> core suite rename (containers#1081), making a local run hard to reproduce from the docs alone. - add a `make claude-agent-acp` target so the claude-code agent works out of the box (supersedes containers#947) - add SUITE/AGENT/MODEL knobs to `make run-evals` for a no-OpenAI-key single-suite run, plus a documented quickstart - fix the default label selector (suite=kubernetes selected zero tasks -> suite=core) and the readiness probe (/health 404 -> /healthz, -f) - rewrite evals/README.md for the ACP agent and document ANTHROPIC_MODEL - refresh the per-suite kubevirt configs and document top-level vs per-suite eval configs Closes containers#1234 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Nuri <marc@marcnuri.com>
Finishes aligning the drifted eval configs with the rest of the tree. - bump the 15 evals/core-eval-testing/** configs from kubernetes-extension@v0.0.3 to @v0.0.4, matching the claude-code, openai-agent, and kubevirt configs - clarify evals/tasks/README.md that a few subdirectories (kiali/scripts, kubevirt/helpers, tekton/helpers) hold shared helpers rather than task scenarios Refs containers#1234 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Nuri <marc@marcnuri.com>
Route the LLM judge through claude-code for the claude-code agent, so the whole flow (agent + judge) runs on the Claude subscription with no OpenAI key. The legacy `verify: contains:` short form is judge-evaluated (semantic), so this is what makes those tasks runnable keyless. Verified locally: list-images-for-pods and ahoy pass keyless with real judge verdicts. - set llmJudge.ref.type: builtin.claude-code in evals/claude-code/eval.yaml (was openai:gpt-5) and the kubevirt/claude-code per-suite config (was no judge / noop). CI is unaffected: it runs evals/openai-agent/eval.yaml, which keeps the independent openai:gpt-5 judge. - pin the adapter with CLAUDE_AGENT_ACP_VERSION (next to MCPCHECKER_VERSION) and mirror the mcpchecker recipe style (set -e + success echo) - document the keyless claude-code judge, the version knob, and that `verify: contains:` is semantic judging rather than a literal match Refs containers#1234, containers#947, containers#995 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Marc Nuri <marc@marcnuri.com>
Member
Author
|
Validated locally end to end (Kind + the MCP server + the Quick local run: make build mcpchecker claude-agent-acp # needs the `claude` CLI installed + signed in
kind create cluster --name mcp-eval-cluster --kubeconfig "$PWD/_output/kubeconfig"
export KUBECONFIG="$PWD/_output/kubeconfig" # export so setup + verify.sh hit the same cluster
make run-server TOOLSETS=core,config
make run-evals SUITE=core AGENT=claude-code MODEL=sonnetWith |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the local
mcpcheckereval workflow reproducible from the docs alone andfully keyless for the Claude Code agent. The docs and Makefile defaults had
drifted since the ACP migration (#839) and the
kubernetes->coresuite rename(#1081), and a local run still needed an OpenAI key for the LLM judge. Docs +
Makefile + eval-config only, no Go/server code touched.
Incorporates the useful parts of #995 and supersedes #947 (the
claude-agent-acpinstall target).
Highlights
Keyless local runs with the Claude Code agent
evals/claude-code/eval.yamlnow judges withbuiltin.claude-codeinstead ofopenai:gpt-5, so withAGENT=claude-codeboth the agent and the LLM judge run onyour Claude subscription. Every suite runs with no OpenAI key, and judge-backed
tasks are really evaluated rather than skipped. (The legacy
verify: contains:short form is semantic LLM judging, not a literal string match, so those tasks need
the judge.) The
kubevirt/claude-codeper-suite config gets the same judge. CI isunaffected: it runs
evals/openai-agent/eval.yaml, which keeps the independentopenai:gpt-5judge.Install + run targets
make claude-agent-acptarget installs the adapter the claude-code agentneeds (supersedes Add/update make target to install deps for mcpchecker agents #947). Idempotent, and version-pinnable via
CLAUDE_AGENT_ACP_VERSION(next toMCPCHECKER_VERSION). CI never installs theadapter, so this knob is purely for local reproducibility.
SUITE/AGENT/MODELknobs onmake run-evals, e.g.make run-evals SUITE=kubevirt AGENT=claude-code MODEL=sonnet. Selects the evalconfig, sets the suite label selector, and exports
ANTHROPIC_MODEL(which drivesboth agent and judge).
run-evalspulls inclaude-agent-acpas a conditionalprereq for the claude-code agent.
Drift / latent-bug fixes
EVAL_LABEL_SELECTORdefaultsuite=kubernetesselected zero tasks after therename -> now
suite=core.make run-serverreadiness probe hit/health(a 404 thatcurl -sswallowed,degrading to a port-open check) -> now
curl -fsS .../healthz.kubernetes-extension@v0.0.4: the 15core-eval-testing/**configs and thekubevirt/openai-agentconfig were staleat
v0.0.3. Also fixed that config's model fromgoogle:gemini-2.0-flashtoopenai:gpt-5so theopenai-agentdirectory actually uses an OpenAI model.Docs
evals/README.mdrewritten for the ACP agent: theclaude-agent-acpprerequisite(it bridges to the
claudeCLI, which runs on your subscription),ANTHROPIC_MODELmodel selection, a keyless single-suite quickstart, top-level vs per-suite configs,
the version knobs, and removal of the dead
examples/...paths, the nonexistentagentbinary reference, the oldcommands:/claude --printblock, and the"OpenAI ACP" mislabel.
evals/tasks/README.mdupdated for the rename(
kubernetes/->core/, addedconfig/helm/tektonfamilies, noted theshared-helper dirs).
Verified locally
Ran the suites end to end (Kind + the MCP server + claude-code on Sonnet, no
OpenAI key):
helm3/3 pass.core23/29 keyless: the 19 deterministic tasks pass, and all 4 judge-backedtasks (
ahoy,debug-app-logs,list-images-for-pods,nodes-top) now passwith the claude-code judge giving real semantic verdicts. They had failed only
because the OpenAI judge had no key, not because the agent was wrong. The
remaining 6 are genuine (1 setup-script break, 5
verify.shmisses), unrelatedto this change.
make run-server's new/healthzprobe confirmed against a live server.CI safety
The mcpchecker CI pipeline is untouched: it drives the eval through
mcpchecker-actionwithevals/openai-agent/eval.yaml(judge unchanged) and nevercalls
make run-evals. The only CI-shared target changed ismake run-server,whose
/healthzprobe is verified (the endpoint returns 200 and is auth-exempt).Not done (follow-ups)
#5); the minimal manualpath is documented in the quickstart instead.
make diff-evalsbaseline naming for per-suite eval configs.Closes #1234