feat(ai-builder): Push agent intent eval cases from data/agents and drop synced case files (no-changelog) by JoseBra · Pull Request #34957 · n8n-io/n8n

JoseBra · 2026-07-25T16:53:34Z

Summary

The agents-team eval corpus now lives fully in the LangTracer agents suite (42 cases: 8 agent-artifact + 34 intent-resolution). This PR makes the tooling and the repo reflect that:

eval:langtracer-push now loads data/agents/ too (previously data/workflows/ only): both directories feed the same upsert planner, --changed detects case edits in either, and a duplicate-slug guard refuses ambiguous selections across the two dirs.
Deletes the 42 case JSONs already synced to LangTracer (verified: push then re-export dry-run reports all 42 unchanged, so the suite copies are byte-equivalent on the compared fields). The seeded carve-out case stays on disk — the case-write API can't represent seeded cases yet.
Repoints pnpm eval:agents to --source langtracer --suite agents --tier agents so the command keeps selecting the intent subset now that the corpus is remote.
README + data/agents/index.ts header updated: both data dirs are authoring scratch (author → calibrate → push → delete); only seeded carve-outs are committed.

How to test

From packages/@n8n/instance-ai with LANGTRACER_URL/LANGTRACER_API_KEY set:

# planner sees both dirs; everything already synced reads as unchanged/none selected
pnpm eval:langtracer-push --suite agents --tier agents --dry-run

Intent cases require a backend with N8N_ENABLED_MODULES=instance-ai,agents to run end-to-end (pnpm eval:agents --base-url ...).

Related Linear tickets, Github issues, and Community forum posts

Follow-up to the TRUST-268 agents suite work.

Review / Merge checklist

I have seen this code, I have run this code, and I take responsibility for this code.
PR title and summary are descriptive. (conventions)
Docs updated or follow-up ticket created.
Tests included.
PR Labeled with Backport to Beta, Backport to Stable, or Backport to v1 (if the PR is an urgent fix that needs to be backported)

🤖 PR Summary generated by AI

🤖 Generated with Claude Code

…rop synced case files (no-changelog) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

n8n-assistant · 2026-07-25T16:54:22Z

PR review overview

Based on ownership of the 46 changed files in this PR:

Ownership	Files owned	Share	Source code	Test files	Misc
@n8n-io/instance-ai	46	100%	+26 / -1,109	+0 / -0	+3 / -3
Total	46	100%	+26 / -1,109	+0 / -0	+3 / -3

codecov · 2026-07-25T16:56:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

cubic-dev-ai

No issues found across 46 files

Architecture diagram

sequenceDiagram
    participant Dev as Developer
    participant CLI as eval CLI
    participant LoadWf as Loader (workflows)
    participant LoadAg as Loader (agents)
    participant Git as Git Repo
    participant LTRClient as LangTracer Client
    participant LTRAPI as LangTracer API

    Note over Dev,LTRAPI: Flow A: eval:langtracer-push (upsert cases)

    Dev->>CLI: pnpm eval:langtracer-push --suite agents --tier agents --dry-run(optional)
    CLI->>LoadWf: loadWorkflowTestCasesWithFiles(filter, exclude)
    LoadWf-->>CLI: workfow cases + fileSlugs
    CLI->>LoadAg: loadAgentEvalTestCasesWithFiles(filter, exclude)
    LoadAg-->>CLI: agent cases + fileSlugs
    CLI->>CLI: Detect duplicate slugs across both dirs
    alt Duplicate slug found
        CLI-->>Dev: Error: duplicate case slug(s)
    else No duplicates
        CLI->>CLI: Apply --tier filter (datasets.includes("agents"))
        alt --changed flag set
            CLI->>Git: git status --porcelain (data/workflows/ + data/agents/)
            Git-->>CLI: modified/new/untracked .json paths
            CLI->>CLI: Intersect changed slugs
        end
        alt --dry-run
            CLI-->>Dev: Print selected cases, stop
        else Real push
            CLI->>LTRClient: upsert suite with case payloads
            LTRClient->>LTRAPI: POST /api/v1/suites/:id/export
            LTRAPI-->>LTRClient: Success
            LTRClient-->>CLI: Upsert results
            CLI-->>Dev: Done
        end
    end

    Note over Dev,LTRAPI: Flow B: eval:agents (run from LangTracer)

    Dev->>CLI: pnpm eval:agents (source langtracer, suite agents, tier agents)
    CLI->>LTRClient: GET suite export (agents)
    LTRClient->>LTRAPI: GET /api/v1/suites/:id/export
    LTRAPI-->>LTRClient: Suite data with 42 cases
    LTRClient-->>CLI: Parsed EvalTestCase array
    CLI->>CLI: Filter by tier "agents"
    CLI-->>Dev: Run evaluations (no local file read)

_{Re-trigger cubic}

github-actions · 2026-07-25T17:02:48Z

Instance AI Discovery Eval ✅

Branch: eval-agents-cases-langtracer-home · Commit: 535f6b37a87240d9814f607899648fca97fb5ea0

Eval output


> @n8n/instance-ai@1.17.0 eval:discovery /home/runner/_work/n8n/n8n/packages/@n8n/instance-ai
> tsx evaluations/discovery/cli.ts --trials 3 --fail-on-zero-pass

Running 11 discovery scenario(s) × 3 trial(s) (model: anthropic/claude-sonnet-4-6, concurrency: 3).

▸ config-evals-skill-loading ... ✓ 3/3 passed (100%)
▸ data-table-natural-list-skill-loading ... ✓ 3/3 passed (100%)
▸ data-table-skill-loading ... ✓ 3/3 passed (100%)
▸ data-table-workflow-skill-loading ... ✓ 3/3 passed (100%)
▸ google-oauth-credential-setup ... ✓ 3/3 passed (100%)
▸ http-node-config-no-browser ... ✓ 3/3 passed (100%)
▸ oauth-with-computer-use-disabled ... ✓ 3/3 passed (100%)
▸ screenshot-dashboard ... ✓ 3/3 passed (100%)
▸ slack-oauth-credential-setup ... ✓ 3/3 passed (100%)
▸ workflow-builder-no-agent-builder-leak ... ✓ 3/3 passed (100%)
▸ workflow-builder-no-credential-ask ... ✓ 3/3 passed (100%)

=== Summary ===
Scenarios: 11/11 above threshold (67%)
Trials: 33/33 passed (100%)
Total time: 1043.6s

github-actions · 2026-07-25T17:09:05Z

Instance AI Workflow Eval

Important

This eval does not re-run on new commits. To test your latest push, re-run it against the PR head:

gh workflow run ci-instance-ai-evals.yml -f pr=34957

…or use the Run workflow button and set pr = 34957.

Caution

🔴 1 of 21 units not green over 3 runs.

Gate: pass@k = 100% (every unit passes at least once across k runs) — 92.1% pass (58/63 trials over 21 units · k=3)

pass@3 95.2% · pass^3 88.5% · LangSmith experiment

Failures (3)

notification-router :: Before building the workflow, the agent established the inco — passed 0/3

The user stated the urgency levels (High/Medium/Low) in their answer, but the payload field name was never established before building — the agent invented urgency as the field name, and the user had to correct it to level in the follow-up turn.

The agent asked about destinations but not about the payload field name; it invented 'urgency' as the field, which the user had to correct in turn 2 — so the payload shape was not established before building.

The agent asked about urgency levels but never asked about the payload field name — it assumed urgency and invented the field, which the user had to correct in Turn 2 to level.

revises-plan-after-rejection/urgent-rows-only — passed 2/3

[builder_issue] The workflow crashed at 'Read Leads Sheet' with 'Sheet with name sheet_1_id not found'. The node's sheetName is configured with mode: 'name' and value: 'sheet_1_id' — a placeholder identifier, not the actual sheet title. The mock's response contained a sheet with title 'Sheet1', but the node looked for a sheet named 'sheet_1_id', which doesn't exist. Because this node errored, all downstream nodes (Only New Rows, Status Is Urgent, Post to #urgent-alerts, Compute Max Row, Update Cursor) never ran, so no Slack posts were made.

revises-plan-after-rejection :: After the rejection, the agent revised the plan to incorpora — passed 2/3

After the user's Turn 2 feedback, the agent in Turn 3 went directly to implementation (workspace_str_replace_file calls) without first presenting a revised textual plan for approval — the user had explicitly asked to see and approve the plan before building, but the agent skipped that step again and relied solely on build-workflow confirmation prompts.

Per-test-case results (7)

Workflow	Status	pass@3	pass^3
`airtable-split-to-slack`	3/3	100%	100%
`notification-router`	3/3	83%	83%
`rest-api-data-pipeline`	3/3	100%	100%
`revises-plan-after-rejection`	3/3	100%	72%
`telegram-chatbot-memory-session`	3/3	100%	100%
`weather-alert`	3/3	100%	100%
`workflow-data-table`	3/3	100%	100%

Workflow checks

Scored over 21 successful build(s). N/A = check did not apply to that workflow. Error = check could not be measured (e.g. judge timeout).

Dimension	Check	Kind	Pass	Fail	Pass rate
`parameter_correctness`	`valid_data_flow`	llm	20	1	95%
`parameter_correctness`	`valid_field_references`	deterministic	20	1	95%
`intent_match`	`fulfills_user_request`	llm	19	2	90%

All workflow checks (3 failing of 34 checks)

Dimension	Check	Kind	Pass	Fail	N/A	Pass rate
`structure`	`has_nodes`	deterministic	21	0	0	100%
`structure`	`has_start_node`	deterministic	21	0	0	100%
`structure`	`has_trigger`	deterministic	21	0	0	100%
`structure`	`no_disabled_nodes`	deterministic	21	0	0	100%
`connection_topology`	`all_nodes_connected`	deterministic	21	0	0	100%
`connection_topology`	`error_routes_consistent`	deterministic	21	0	0	100%
`connection_topology`	`handles_multiple_items`	llm	21	0	0	100%
`connection_topology`	`no_unreachable_nodes`	deterministic	21	0	0	100%
`connection_topology`	`switch_fallback_output_enabled`	deterministic	3	0	18	100%
`parameter_correctness`	`correct_node_operations`	llm	21	0	0	100%
`parameter_correctness`	`expressions_reference_existing_nodes`	deterministic	5	0	16	100%
`parameter_correctness`	`google_sheets_rlc_default_mode`	deterministic	3	0	18	100%
`parameter_correctness`	`http_generic_auth_type_matches_prompt`	deterministic	0	0	21	—
`parameter_correctness`	`item_flow_independent_source_execute_once`	deterministic	0	0	21	—
`parameter_correctness`	`item_flow_paired_item_references`	deterministic	0	0	21	—
`parameter_correctness`	`no_empty_set_nodes`	deterministic	1	0	20	100%
`parameter_correctness`	`no_invalid_from_ai`	deterministic	0	0	21	—
`parameter_correctness`	`valid_data_flow`	llm	20	1	0	95%
`parameter_correctness`	`valid_field_references`	deterministic	20	1	0	95%
`parameter_correctness`	`valid_node_config`	deterministic	21	0	0	100%
`intent_match`	`fulfills_user_request`	llm	19	2	0	90%
`communication`	`response_describes_changes_accurately`	llm	21	0	0	100%
`ai_nodes`	`agent_has_dynamic_prompt`	deterministic	3	0	18	100%
`ai_nodes`	`agent_has_language_model`	deterministic	3	0	18	100%
`ai_nodes`	`memory_properly_connected`	deterministic	3	0	18	100%
`ai_nodes`	`memory_session_key_expression`	deterministic	3	0	18	100%
`ai_nodes`	`tools_have_parameters`	deterministic	0	0	21	—
`ai_nodes`	`vector_store_has_embeddings`	deterministic	0	0	21	—
`nodes_craftsmanship`	`code_node_no_http_requests`	deterministic	7	0	14	100%
`nodes_craftsmanship`	`descriptive_node_names`	llm	21	0	0	100%
`nodes_craftsmanship`	`no_unnecessary_code_nodes`	llm	21	0	0	100%
`efficiency`	`no_excessive_build_failures`	deterministic	21	0	0	100%
`security`	`inbound_trigger_auth_defaults`	deterministic	3	0	18	100%
`security`	`no_hardcoded_credentials`	deterministic	9	0	12	100%

feat(ai-builder): Push agent intent eval cases from data/agents and d…

b283c64

…rop synced case files (no-changelog) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

JoseBra requested review from a team and schrothbn and removed request for a team July 25, 2026 16:53

n8n-assistant Bot added the cla-signed label Jul 25, 2026

JoseBra marked this pull request as ready for review July 25, 2026 16:58

JoseBra requested a review from bjorger July 25, 2026 16:59

n8n-assistant Bot added the n8n team Authored by the n8n team label Jul 25, 2026

cubic-dev-ai Bot reviewed Jul 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ai-builder): Push agent intent eval cases from data/agents and drop synced case files (no-changelog)#34957

feat(ai-builder): Push agent intent eval cases from data/agents and drop synced case files (no-changelog)#34957
JoseBra wants to merge 1 commit into
masterfrom
eval-agents-cases-langtracer-home

JoseBra commented Jul 25, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

n8n-assistant Bot commented Jul 25, 2026

Uh oh!

codecov Bot commented Jul 25, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

github-actions Bot commented Jul 25, 2026

Uh oh!

github-actions Bot commented Jul 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

JoseBra commented Jul 25, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to test

Related Linear tickets, Github issues, and Community forum posts

Review / Merge checklist

Uh oh!

n8n-assistant Bot commented Jul 25, 2026

PR review overview

Uh oh!

codecov Bot commented Jul 25, 2026

Codecov Report

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 25, 2026

Instance AI Discovery Eval ✅

Uh oh!

github-actions Bot commented Jul 25, 2026

Instance AI Workflow Eval

Workflow checks

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JoseBra commented Jul 25, 2026 •

edited by cubic-dev-ai Bot

Loading