Skip to content

fix(eval): reuse the Framework checkout in the PAI-Bench-C launcher - #315

Open
salignatmoandal wants to merge 1 commit into
NVIDIA:mainfrom
salignatmoandal:fix/paibench-c-framework-paths
Open

salignatmoandal wants to merge 1 commit into
NVIDIA:mainfrom
salignatmoandal:fix/paibench-c-framework-paths

Conversation

@salignatmoandal

Copy link
Copy Markdown

The PAI-Bench-C launcher exits before setup from its current .agents/skills/evaluation/ location because it only recognizes the legacy cookbooks/cosmos3 root. Even after recognizing Framework, its checkout selection would otherwise fall back to a nested clone under packages/.

Recognize the containing Framework checkout by pyproject.toml and cosmos_framework/, and reuse it before considering legacy nested checkouts. Preserve explicit COSMOS3_REPO overrides, clone destinations, and legacy cookbook layouts. Document checkout selection and the command for the new CPU path tests.

Validation:

  • Before the fix, three path tests failed on the Framework layout; the legacy and invalid-root cases passed.
  • All seven CPU tests now pass: current layout, spaces and unrelated working directory, containing-checkout precedence, explicit checkout and clone overrides, legacy checkouts and clone fallback, and invalid-root rejection before setup.
  • python3 -m unittest discover -s .agents/skills/evaluation/cosmos3/generator/paibench_c -p test_launcher_paths.py -v
  • bash -n .agents/skills/evaluation/cosmos3/generator/paibench_c/run_paibench_c.sh
  • Ruff lint and format checks on the new test, and git diff --check, passed.

The tests exercise the actual bootstrap section before setup side effects; the invalid-root case executes the complete script. No model downloads, GPU inference, or benchmark scoring were performed. Shard counting and evaluation coverage remain outside this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant