Repository navigation
fix(bench): keep regrade copies faithful and mirror grader availability - #622
Merged
Merged
Conversation
…ty (#614) Review follow-ups on #621: copytree now preserves agent-written symlinks as links instead of dereferencing outside targets into --out; a re-grade that hits an unavailable check records the grader-unavailable reason and invalid status like a fresh run (the reason is still stripped when unavailability resolves); same-named input directories fail upfront instead of dying mid-copy; and the docs note that trials without a parseable contract result.json are skipped.
github-project-automation
Bot
moved this from Backlog
to Done
in Make Workshop Great Again
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up fixes to
regrade(merged as #621, for #614 / SPEC-414 REQ-007) from post-merge review:copytree(..., symlinks=True)— an agent-written symlink in a stored trial workspace was dereferenced during the copy, pulling the outside target's contents recursively into--out. Links now stay links.grader-unavailablereason was only ever removed (when a re-grade resolved it), never added (when a re-grade newly hit an unavailable check). A freshrun_trialrecordsstatus: invalid+ the reason in this case; a re-grade now mirrors both directions so downstream report rates treat it identically.<out>/<base.name>/collided for same-named input dirs with aFileExistsErrormid-copy, leaving a partial--out. Now aSystemExitupfront.regrade's docstring and the howto now state that only trials with a parseable contractresult.jsonare copied; incomplete trial dirs are skipped (matchingbench_report.load).Test plan
RegradeTest7/7 pass locally — three new tests: symlink stays a link, unavailability mirrors both directions, same-named inputs fail before any copypython3 -m py_compileclean;git diff --checkcleantest_agent_bench.py— 2 remaining failures are environmental (no oracle toolchain / missingpiadapter locally;ana-paintball-wspasses), identical to the pre-change baseline