fix(evals): keep ambient Jev key out of release hosts - #132
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Independent current-head verification at All seven GitHub checks are green. Six focused tests pass. A real ordinary OpenCode evaluation child and a fake canary child both omit a synthetic ambient |
Why
A paid Flow release run authorized only for OpenAI inherited the local
TYPESAFE_API_KEY. The plugin enables Jev shadow advice when that key is present, so an ordinary evaluation or exact-artifact canary could make an unapproved TypeSafe request. The campaign was stopped with zero scored attempts after the evaluation-host leak was reproduced.Scope
Remove the ambient TypeSafe key from ordinary
EvalHostchildren and the maintained canary launcher. Preserve explicit key injection for the separate livemanager-plus-jevtreatment. Add two model-free child-process regressions with synthetic keys. They check the actual child environment, confirm unrelated values remain, and confirm the canary consumes its paid dispatch.Tradeoffs
The EvalHost child-environment assertion uses Linux
/procand skips elsewhere. The canonical release campaign runs on Linux. The fake canary executable test skips Windows. Existing live-treatment tests cover intentional Jev-arm credential injection.Blast Radius
Evaluation and canary child environments change. The production plugin still defaults to bounded Jev shadow advice when a user configures a TypeSafe key. This PR does not rerun the stopped campaign, alter its ledger, or authorize new paid calls. TypeSafe usage from the interrupted partial attempt is unknown because no attempt transcript was retained.
Verification
The ordinary real-child test failed before the fix with
Expected false, Received true, then passed. The canary fake-child test sawpresentbefore its fix andabsentafterward. Full push preflight passed with 1,591 tests and no failures. The original EvalHost reproduction reportsambientTypeSafeKeyInChild: false.bun run replaymatched 13 of 13 cassettes, and the model-free OpenCode 1.18.31 packed-host smoke passed all 22 checks. No paid calls ran for this PR.