Skip to content

fix(evals): keep ambient Jev key out of release hosts - #132

Merged
vriesd merged 4 commits into
mainfrom
fix/eval-strip-ambient-jev-key
Sep 27, 2026
Merged

vriesd merged 4 commits into
mainfrom
fix/eval-strip-ambient-jev-key

Conversation

@vriesd

@vriesd vriesd commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Why

A paid Flow release run authorized only for OpenAI inherited the local TYPESAFE_API_KEY. The plugin enables Jev shadow advice when that key is present, so an ordinary evaluation or exact-artifact canary could make an unapproved TypeSafe request. The campaign was stopped with zero scored attempts after the evaluation-host leak was reproduced.

Scope

Remove the ambient TypeSafe key from ordinary EvalHost children and the maintained canary launcher. Preserve explicit key injection for the separate live manager-plus-jev treatment. Add two model-free child-process regressions with synthetic keys. They check the actual child environment, confirm unrelated values remain, and confirm the canary consumes its paid dispatch.

Tradeoffs

The EvalHost child-environment assertion uses Linux /proc and skips elsewhere. The canonical release campaign runs on Linux. The fake canary executable test skips Windows. Existing live-treatment tests cover intentional Jev-arm credential injection.

Blast Radius

Evaluation and canary child environments change. The production plugin still defaults to bounded Jev shadow advice when a user configures a TypeSafe key. This PR does not rerun the stopped campaign, alter its ledger, or authorize new paid calls. TypeSafe usage from the interrupted partial attempt is unknown because no attempt transcript was retained.

Verification

The ordinary real-child test failed before the fix with Expected false, Received true, then passed. The canary fake-child test saw present before its fix and absent afterward. Full push preflight passed with 1,591 tests and no failures. The original EvalHost reproduction reports ambientTypeSafeKeyInChild: false. bun run replay matched 13 of 13 cassettes, and the model-free OpenCode 1.18.31 packed-host smoke passed all 22 checks. No paid calls ran for this PR.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-26T23:56:06.156354Z f7bdc1c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vriesd vriesd changed the title fix(evals): keep ambient Jev key out of ordinary hosts fix(evals): keep ambient Jev key out of release hosts Sep 27, 2026
@vriesd

vriesd commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Independent current-head verification at fba4f3d231571b3259926d5bb3757af199c86602: PASS.

All seven GitHub checks are green. Six focused tests pass. A real ordinary OpenCode evaluation child and a fake canary child both omit a synthetic ambient TYPESAFE_API_KEY while retaining unrelated environment values. The fake canary also consumes exactly one local paid-dispatch slot. The explicit live Jev treatment still reinjects its key only for the manager-plus-jev arm. The branch is clean and mergeable. No paid provider call ran in this review.

@vriesd
vriesd merged commit b29a7b4 into main Sep 27, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants