Skip to content

chore: update mcpchecker evaluation results#1286

Merged
Cali0707 merged 1 commit into
mainfrom
chore/update-eval-results
Jul 13, 2026
Merged

chore: update mcpchecker evaluation results#1286
Cali0707 merged 1 commit into
mainfrom
chore/update-eval-results

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Automated Evaluation Results Update

This PR updates the mcpchecker evaluation results from the weekly scheduled run.

Run details:


This PR was automatically generated by the mcpchecker workflow.

@github-actions
github-actions Bot requested review from Cali0707 and manusa as code owners July 13, 2026 12:58

@manusa manusa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

=== Evaluation Results ===
Results file: mcpchecker-openai-agent-kubernetes-test-out.json
=== Evaluation Summary ===

  ✓ ahoy (assertions: 3/3)
  ✓ create-canary-deployment (assertions: 3/3)
  ✓ create-network-policy (assertions: 3/3)
  ✓ create-nginx-pod (assertions: 3/3)
  ✓ create-pod-mount-configmaps (assertions: 3/3)
  ✓ create-pod-resources-limits (assertions: 3/3)
  ✓ create-simple-rbac (assertions: 3/3)
  ✓ debug-app-logs (assertions: 3/3)
  ✓ deployment-traffic-switch (assertions: 3/3)
  ✓ filter-events-by-type (assertions: 3/3)
  ✓ filter-namespace-by-name (assertions: 3/3)
  ✓ fix-crashloop (assertions: 3/3)
  ✓ fix-image-pull (assertions: 3/3)
  ✓ fix-pending-pod (assertions: 3/3)
  ~ fix-probes (assertions: 2/3)
      - MaxToolCalls: Too many tool calls: expected <= 20, got 26
  ✓ fix-rbac-wrong-resource (assertions: 3/3)
  ~ fix-service-routing (assertions: 2/3)
      - MaxToolCalls: Too many tool calls: expected <= 20, got 33
  ✓ fix-service-with-no-endpoints (assertions: 3/3)
  ✓ horizontal-pod-autoscaler (assertions: 3/3)
  ✓ list-images-for-pods (assertions: 3/3)
  ✓ multi-container-pod-communication (assertions: 3/3)
  ✓ nodes-top (assertions: 3/3)
  ✓ resize-pvc (assertions: 3/3)
  ✓ rolling-update-deployment (assertions: 3/3)
  ✓ scale-deployment (assertions: 3/3)
  ✓ scale-down-deployment (assertions: 3/3)
  ~ setup-dev-cluster (assertions: 2/3)
      - MaxToolCalls: Too many tool calls: expected <= 20, got 44
  ✓ ssa-field-preservation (assertions: 3/3)
  ✓ statefulset-lifecycle (assertions: 3/3)

Tasks:      29/29 passed (100.00%)
Assertions: 84/87 passed (96.55%)
Tokens:     ~1612835 (estimate - excludes system prompt & cache)
MCP schemas: ~96686 (included in token total)
Agent used tokens:
  Input:  269997 tokens
  Output: 143990 tokens
Judge used tokens:
  Input:  7663 tokens
  Output: 1381 tokens
=== Threshold Verification ===

Task Pass Rate:      100.00% >= 80.00% ✓
Assertion Pass Rate: 96.55% >= 80.00% ✓

Result: PASSED

🎉

@manusa manusa added this to the 0.1.0 milestone Jul 13, 2026
@Cali0707

Copy link
Copy Markdown
Collaborator

100% pass rate - nice fixes @manusa !

@Cali0707
Cali0707 merged commit 550dcf0 into main Jul 13, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants