Skip to content

fix(backtests): require sign-in and bill ATL Credits for hosted runtimes - #705

Merged
FlyM1ss merged 5 commits into
mainfrom
fix/hosted-runtime-sign-in
Oct 31, 2026
Merged

FlyM1ss merged 5 commits into
mainfrom
fix/hosted-runtime-sign-in

Conversation

@FlyM1ss

@FlyM1ss FlyM1ss commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Guests could start AI Hedge Fund backtests billed to the operator's OPENROUTER_API_KEY with no Credits reservation. This PR closes that and charges every AI Hedge Fund run to the caller's ATL Credits.

Sign-in

  • POST /backtest/run refuses a guest on any non-pipeline runtime with a 401. The check runs before the window bound and the readiness checks, so a guest learns nothing about how the runtime is configured.
  • The agent ownership check runs before that 401. A stranger's agent gets the same 403/404 as any agent id, so the 401 can't reveal that an agent is a hosted one.
  • The check is keyed on != PIPELINE, so TradingAgents (TradingAgents Hosted Runtime Integration #676) is covered too.
  • The modal stops a guest's hosted launch with a sign-in message before the agent and progress panel start.

Credits (AI Hedge Fund)

  • The route rejects billing_mode=byok with a 422, because the model is platform-managed.
  • The route signs a platform-credits handoff naming the managed OpenRouter model. It admits the run through the same _admit_platform_credits as a pipeline run, priced at its ceiling: one decision per trading session, each at the per-decision ceiling. The slot ledger therefore counts AI Hedge Fund and pipeline Credits runs against each other, and they can't each pass alone and then starve together. A short balance gets a 402, a restricted account a 403.
  • The child meters each whole decision, because upstream calls its model inside its own interpreter. Each decision reserves a ceiling of (analysts + 1) × tickers calls at 8k input / 2k output tokens. Reservations go through reserve_platform_credits, which is now shared with LLMExecutionService.
  • bridge.py meters every model call with an inheritable langchain-core callback hook and reports tokens plus calls and unmetered_calls. Counting calls is what separates "no model call" from "calls whose usage went missing".
  • Each decision settles at the reported tokens. Usage above the ceiling is capped at the ceiling, not recorded as overage.
  • The hold is released only when nothing was spent: zero calls reported, or a bridge that left no file at all.
  • Every unknown spend settles at the ceiling:
    • unmetered calls, which log ERROR: ai_hedge_fund.usage_unmetered;
    • a completed decision with no report, which logs ERROR: ai_hedge_fund.usage_unreported;
    • a timeout;
    • a failed exit that left the started marker but no readable usage (an OOM SIGKILL, a crash mid-write);
    • any other escape from the decision, including the SystemExit that a cancel's SIGTERM raises inside subprocess.run.
  • The parent relays ERROR: ai_hedge_fund. lines to the service log live, like ERROR: llm. lines.
  • An empty balance mid-run stops the run with INSUFFICIENT_CREDITS, same as a pipeline run. finalize_run releases only a hold left by a SIGKILLed child. The bridge shares the child's process group, so it dies with the child.
  • Run metadata records billing_mode/provider_id, and the modal's launch config says platform_credits, so the results panel shows "ATL Credits" instead of "—".
  • The Docker build fails if the venv's langchain-core lacks the callback-hook API.
  • The run modal now says the run is "billed to your ATL Credits". The app.js cache-buster is bumped to v=164.

Verification

  • I ran the meter under upstream's pinned stack (langchain-core 0.3.86, langgraph 0.2.56). A LangGraph parallel fan-out, a runnable batch and a ContextThreadPoolExecutor were all counted on worker threads, a call without usage_metadata was counted as unmetered, and calls outside the scope were not counted.
  • Billing tests run against a real SQLite Credits store. They cover SIGTERM-style interrupts, unmetered calls, zero-call decisions and malformed reports.
  • Full suite: the only local failures are the known reportlab and mcp version ones, which also fail on main.

Not in this PR: metering each model call at an ATL-side OpenAI-compatible endpoint backed by LLMExecutionService. That would remove the ceiling heuristic and keep OPENROUTER_API_KEY out of the upstream process. It's a redesign and belongs in its own PR.

Prod: MAX_AI_HEDGE_FUND_TRADING_DAYS=0 is still set on Render, so AI Hedge Fund answers 503. Set it back to 10 after this deploys.

Docs: six docs/source/lab/ pages (accounts, credits_billing, getting_started, key_features, marketplace, operating_modes) now say AI Hedge Fund needs sign-in and bills ATL Credits. credits_billing gains an "AI Hedge Fund runs" section.

🤖 Generated with Claude Code

POST /backtest/run only 401'd guests on the pipeline runtime. A guest
(X-Session-Id alone) could clone the AI Hedge Fund agent and launch a
run billed to the operator's OPENROUTER_API_KEY with no Credits
reservation; the per-owner slot and rate-limit key were a session id
the guest mints itself.

Refuse a guest on any non-pipeline runtime, before the window bound and
readiness checks, so a guest cannot learn whether a runtime is armed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agentic-trading-lab Ready Ready Preview Oct 31, 2026 2:27am UTC

Upstream calls its model inside its own interpreter, so ATL meters whole
decisions: the route signs a platform-credits handoff (BYOK is a 422, a
balance that cannot hold one decision a 402), and the child reserves a
per-decision ceiling and settles at the token usage the bridge reports via
langchain-core's get_usage_metadata_callback. Usage above the ceiling is
capped, never recorded as overage. Unreported usage bills the ceiling on a
completed or timed-out decision and is released on a crash before the
usage scope opened. The existing finalize_run releases holds a killed
child leaves, and the timeout card reads the spend like a pipeline run's.

The Docker build now fails if the venv's langchain-core lacks the meter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss FlyM1ss changed the title fix(backtests): require sign-in for hosted-runtime backtests fix(backtests): require sign-in and bill ATL Credits for hosted runtimes Oct 31, 2026
A failed decision with no usage report was released, so an OOM SIGKILL
after model calls (or an output write that raised outside the handler)
made the decision free. The bridge now writes a started marker before
upstream can call the model and its final output inside the try; the
adapter bills the ceiling for any failure past the marker without a
readable report, and for a clean exit whose output it cannot read. Only
a bridge that left no file at all, so made no call, is released, and
billing treats any outcome other than "failed" as billable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Bill the ceiling on every escape from a decision, SystemExit from a
  cancel's SIGTERM included, instead of leaving the hold for finalize_run
  to release for free.
- Meter calls, not just tokens: the bridge's own inheritable callback
  hook reports calls and unmetered_calls, so calls whose usage went
  missing bill the ceiling (ERROR, relayed live) and only zero calls is
  free.
- Admit AI Hedge Fund runs through the shared Credits admission at their
  window ceiling, so they enter the in-flight ledger with pipeline runs.
- Share the reserve-error mapping with LLMExecutionService.
- Run the ownership check before the guest 401, so it cannot reveal that
  a stranger's agent is hosted.
- Record the Credits lane on hosted runs and in the modal's launch
  config, and stop a guest's hosted launch client-side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Six user-facing pages still said the hosted runtime needed no sign-in and
spent no Credits. Add an "AI Hedge Fund runs" section to Credits & Billing:
per-decision holds, full-hold charge when usage is unknown or a decision is
cut short, and the whole-window launch check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss
FlyM1ss merged commit a3aa0f0 into main Oct 31, 2026
10 checks passed
@FlyM1ss
FlyM1ss deleted the fix/hosted-runtime-sign-in branch October 31, 2026 02:42
computedocean pushed a commit to computedocean/AgenticTrading that referenced this pull request Oct 31, 2026
Open-Finance-Lab#704 and Open-Finance-Lab#705 each fit under the ceiling against their own base;
merged together they put main at 40,004. Drop one redundant word
(39,994).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — f818cf9d Deployed Oct 31, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant