Repository navigation
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 14edc012-44cd-42a2-bea7-ae4041bcc54e
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Missing dependency links in the restored checkout and overly broad bundle discovery can prevent workflow recovery.
2 open issues
What changed in this PR
Builds on portable result bundles to retry benchmark and experiment execution errors in place, preserving completed trials and reporting the full evaluation.
Changes:
- Adds retry commands, shared recovery logic, and regression tests.
- Adds a manual recovery workflow and deployment triggers for repaired results.
- Documents recovery requirements and updates repository run helpers.
| File | Description |
|---|---|
script/​run-experiment.sh |
Adds experiment retry mode. |
script/​run-benchmark.sh |
Adds benchmark retry mode. |
script/​retry-evaluation.sh |
Locates a bundle and invokes recovery. |
packages/​agent-eval/​src/​trial/​run.ts |
Preserves saved outcomes during artifact cleanup. |
packages/​agent-eval/​src/​trial/​run.test.ts |
Tests outcome preservation and cleanup. |
packages/​agent-eval/​src/​cli/​retry.ts |
Implements selective retries and full reporting. |
packages/​agent-eval/​src/​cli/​retry.test.ts |
Covers recovery, preservation, and validation. |
packages/​agent-eval/​src/​cli/​retry-script.test.ts |
Tests workflow helper arguments and discovery. |
packages/​agent-eval/​src/​cli/​commands/​experiment.ts |
Registers the experiment retry command. |
packages/​agent-eval/​src/​cli/​commands/​benchmark.ts |
Registers the benchmark retry command. |
packages/​agent-eval/​docs/​references/​trials-and-results.md |
Explains recovery and comparison eligibility. |
packages/​agent-eval/​docs/​references/​plans.md |
Documents retrying merged plans. |
packages/​agent-eval/​docs/​references/​cli.md |
Documents retry commands, options, and constraints. |
packages/​agent-eval/​docs/​AGENTS.md |
Adds recovery guidance for agents. |
docs/​workflow-recovery.md |
Explains workflow and local recovery procedures. |
docs/​experiments.md |
Updates experiment CLI guidance. |
docs/​benchmarks.md |
Updates benchmark CLI guidance. |
.github/​workflows/​retry-evaluation.yml |
Adds manual artifact recovery and republication. |
.github/​workflows/​deploy.yml |
Triggers deployment after recovery workflows. |
.changeset/​chatty-papayas-obey.md |
Records the new retry commands. |
Review effort: Balanced
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| ref: ${{ steps.source.outputs.revision }} | ||
| path: evaluation-source |
| if [[ -d "$results_directory" ]]; then | ||
| while IFS= read -r -d '' output; do | ||
| outputs+=("$output") | ||
| done < <(find "$results_directory" -type f -name output.json -print0) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #453.
Add dedicated
benchmark retryandexperiment retrycommands to recover execution errors in an existing result bundle. Retry preserves completed trial files, trial IDs, and the original manifest, then reports the full evaluation rather than only the retried subset.Add a manual retry evaluation workflow that downloads a merged benchmark or experiment artifact, restores its original evaluation definitions, and runs the same retry command. Repaired benchmark artifacts retain the original result date and are included in deploy.
The workflow publishes a new complete artifact rather than modifying the original run's artifact. Use the repair run ID for subsequent attempts. Recovery requires a merged result bundle; jobs that stopped before producing results still need to finish first.
Changelog
New
benchmark retryandexperiment retrywith existing directory, concurrency, and token options.retrymode to the repository's benchmark and experiment run helpers.Changed
Removed