Skip to content

feat(effect-ai): integration, catalog sweep and release readiness (phase 8) - #99

Merged
anandpant merged 8 commits into
mainfrom
effect/phase-8
Nov 20, 2026
Merged

anandpant merged 8 commits into
mainfrom
effect/phase-8

Conversation

@anandpant

@anandpant anandpant commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Phase 8 of packages/foundry-effect-ai/docs/PLAN.md: integration and release readiness, plus the reviewer's handoff items.

  • Integration. src/internal/notImplemented.ts is deleted; no stub paths remain, and test/matrix.ts has no todos (62 contract tests). Tree-shaking budgets are re-measured at measured size plus about 20%, and a module without a budget now fails instead of falling back to 64 kB.
  • LIVE_MODELS for the Effect live suite, as in packages/foundry-ai: all, provider names or catalog IDs; an unknown token fails. Unset, the run is unchanged (50 cases on the default models). Each language cell file exports cellsFor(model), which runs the shared cells with that model's settings: adaptive thinking (with display: 'summarized') or a budget for Claude, no reasoning settings for non-reasoning models, and no effort for Grok 4.20 reasoning. A sweep never records, and FOUNDRY_RECORD with LIVE_MODELS fails. Vitest 5 joins suite and test with > for -t, which the docs now show.
  • transformClient sees failures (review item 1). HttpClient.filterStatusOk runs on the authorized client before transformClient, as upstream OpenAiClient does, so retries and taps see 429 and 5xx. send still checks the final status, so a transform that hands back a failed response gets the mapped error rather than a decoded body. The contract-only transform-client cell covers both: HttpClient.retry over a synthetic 503 then 200 makes two requests and succeeds, and a pass-through transform's 400 fails json and sse with InvalidRequestError. Each half failed before its fix.
  • Recording keeps error fixtures. With the filter in place, the live recorder also captures responses that reach transformClient as StatusCodeError, and writeFixtures refuses an empty recording. Re-recording error-mapping against Foundry kept all eight responses with identical bodies and added their golden requests, which the old fixture lacked.
  • Typed metadata keys (review item 2). src/internal/metadata.ts augments effect/ai/AiError (rate limit, quota, authentication, invalid request, internal provider, unknown), Response part metadata, Prompt part options and DecideResponseMetadata. Only our keys are added (foundry and foundry-<vendor>), and each is declared once. FoundryClient imports the file, so every public module's emitted .d.ts carries it. The keys are typed under exactOptionalPropertyTypes only: a conditional base adds nothing without the flag, because an optional key would then admit undefined, which Effect's JSON index signatures reject (TS2411). test/types.ts asserts the typed reads. The tree-shaking consumer typecheck now runs in both modes: every module alone and all eight together, plus typed reads in exact mode. Removing the conditional base fails the non-exact run with TS2411; removing the import fails the typed reads.
  • Nits. optionalHeaders is gone; setHeaders drops undefined values. json still builds its decoder per call: callers pass the schema per call, so caching it isn't a trivial change, and the inner parser is already memoized.
  • Adapter fixes the sweep found.
    • xAI error bodies are { code, error: "<message>" }; the message was dropped. Regression fixture recorded from the live 400.
    • Anthropic adaptive thinking takes display: 'summarized' | 'omitted'. Opus 4.7 and later, Claude 5 and 5.5 omit thinking text without it, and they reject budget thinking.
    • The Grok config doc now notes that Grok 4.20 reasoning rejects effort.
  • Docs. New package README (install, configuration, one example per module, the per-module import rule, typed metadata) with absolute links, since the docs aren't in the tarball. It states that retryTransient covers 408, 429, 500, 502, 503 and 504. New docs/README.md with the sweep tables, the Foundry quirks the adapters handle, and what's out of scope. PLAN.md is marked done and kept as the design record. The root README no longer says "in development".

Sweep results

LIVE_MODELS=all pnpm run test:live:effect: 600 cases, 515 passed on the first run. After isolated reruns of every failure, plus a full Anthropic re-sweep and Grok 4.20 reruns on the final settings, 539 of 600 pass. All 61 remaining failures are outside the adapter:

  • 42: Foundry or the vendor refused the route or model.
    • The four Codex models have no Chat Completions backend.
    • GPT-6 Astra and GPT-6.1 Sol on Chat reject tools while reasoning, and they also reject reasoning_effort: 'none'. Adding them to the 'none' flag was tried, didn't help, and was reverted.
    • xAI returns 404 for grok-build-0-1 on this account.
  • 19: model behavior, failed the same way on rerun.
    • GPT-4.1 models and Gemini 3 Flash answer "Hi" to the one-word hello prompt.
    • GPT-4.1 nano calls the tool twice; GPT-4.1 and GPT-5.1 sometimes skip it.
    • GPT-6 Astra and 6.1 Sol return empty reasoning summaries, and Claude Opus 4.7 and Sonnet 5.5 don't think, on the reasoning puzzle.

Embeddings (3 models), decisions (GPT-6 Luna) and the transport cells all pass. Full tables are in packages/foundry-effect-ai/docs/README.md.

Checks

  • pnpm run verify passes.
  • pnpm run test:live:effect on the defaults: 48 of 50 passed. GPT-6 Luna reasoning (empty summary) and Grok 4.6 json-tools (skipped the tool) are the known flakes; both passed on an isolated rerun.
  • The README examples were typechecked and run against Foundry: text, streaming, tools over two turns, structured output on Claude Sonnet 5.5, decisions and embeddings.
  • Fixtures and docs contain no stack host or account IDs.

Notes for review

  • Without exactOptionalPropertyTypes, our metadata keys read as plain JSON; consumers need no compiler flag to build.
  • In a sweep, the replay check requires reasoning only when the model produced some. Adaptive Claude models sometimes don't think before an obvious call, and the second turn already proves the replay works. The recorded defaults keep the strict check, so a recording without reasoning still fails.
  • Not done here: the release itself (pnpm run release foundry-effect-ai), npm and merging.

@anandpant anandpant changed the title feat(effect-ai): LIVE_MODELS sweep, filterStatusOk before transformClient, drop stub feat(effect-ai): integration, catalog sweep and release readiness (phase 8) Nov 20, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T04:14:43.475108Z 42a2f13 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc7e5f5a81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

anandpant commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Merge activity

  • Nov 20, 4:15 AM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Nov 20, 4:15 AM UTC: @anandpant merged this pull request with Graphite.

@anandpant
anandpant merged commit d06ef0f into main Nov 20, 2026
8 checks passed
@anandpant
anandpant deleted the effect/phase-8 branch November 20, 2026 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant