Skip to content

fix(backtests): stop reading LLM confidence off the 0-1 scale - #674

Merged
FlyM1ss merged 6 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-confidence-range
Oct 30, 2026
Merged

FlyM1ss merged 6 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-confidence-range

Conversation

@subidhkhanal

@subidhkhanal subidhkhanal commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

The bug

make_trading_decision_with_llm (domain/backtesting/portfolio_manager.py) reads each action's confidence with no range check. Confidence is meant to be 0–1. A model that answers on a 0–100 scale (25 meaning 25%):

  • clears the 0.3 floor (25 < 0.3 is false), although 25% is below it;
  • multiplies the confidence-sized buy by 100 when no position_size is given (risk_amount = 2% of equity × confidence).

Reproduced on main with {"action": "buy", "symbol": "AAPL", "confidence": 25} on a $100k account:

Processing: AAPL (BUY, conf=2500%)
✅ BUY AAPL: 500 shares @ $100.00 (conf: 2500%)

That is $50k, half the account. A correct 0.25 would have been skipped by the floor. Other values on main:

  • 1.5 bought at "150%";
  • NaN and Infinity both traded;
  • 80 was dropped only because the inflated buy needed $160k of cash.

The fix

Before the 0.3 floor, an action whose confidence is a number outside [0, 1], or is not finite, is skipped and logged:

⚠️  AAPL: Skip (confidence 25 outside 0-1)
  • This is the same 0–1 rule validate_confidence in infrastructure/llm/validator.py already enforces for external agents.
  • No clamping and no ÷100 rescaling. A model's verbalized confidence scale is not reliable enough to guess from (arXiv 2412.14737, ICLR 2025), and clamping would turn a nonsense value into a confident one.
  • Only that action is skipped; the run is not aborted.

Unchanged:

Tests

New cases in tests/backtesting/test_portfolio_manager_move.py:

  • 25, 80 and 1.5 → no action, plus the skip line;
  • -0.5, NaN and Infinity → no action, plus the skip line;
  • 0.3, 1.0 and 1 → the buy still goes through.

The out-of-range cases failed on main as listed above and pass with the fix. Focused suites (tests/backtesting, tests/domain/backtesting, tests/infrastructure/llm, tests/conformance): 1140 passed, 0 failed.


Maintainer follow-up

The sections above describe the branch's original commit. The title changed from "skip an LLM action whose confidence is outside 0-1" because the branch no longer skips every such action.

#673 merged first and already skips any action whose confidence is outside 0–1 (read_confidence), so this branch's own hunk was a duplicate. The merge of main keeps #673's version. The commits after it narrow that rule to where confidence actually drives the outcome, and make a run that breaks it visible.

Gate (LLM_GATE_VERSION = 2). A confidence that is not a number from 0 to 1 (25, NaN, "high", a bool) is not read at all. It neither sizes a buy nor decides the 0.3 floor.

  • A sell (always the sellable position) or a buy with its own position_size trades as written. On main these were dropped, so a model on a 0–100 scale could never exit a position. Pipeline prompts name no scale.
  • A buy that would be sized from the confidence is skipped. A strict run rejects the batch instead, as it does a bad symbol, so a billed run stops on the first bar rather than charging for a run that never buys.
  • This partly reverses fix(backtests): read every LLM action's confidence by one 0-1 rule #673's rule, and its pipeline test is updated to match.

Visible, not a flat curve.

  • One flushed ERROR: llm.unusable_confidence line per run, which the parent relays to the service log.
  • Counts in agent_runs.metadata.unusable_confidence, from the engine and the leaderboard. The live board adds them across checkpoints.
  • The H6 guard (and the live board's early stop) takes confidence_skipped_steps off its numerator. llm_decisions is untouched, because the decision tape reads an unchanged count as a fallback bar.

Provenance.

  • llm_gate_version is recorded on every LLM run.
  • It is part of the repeat-sample pooling key.
  • Cached leaderboard rows recorded under another version print a once-per-process warning, mirroring the feed-drift one. There is no auto-refresh, since a recompute from a public GET would be a billable deploy.

Same bug elsewhere.

  • The Trading Algo loop (backtest_custom_algo.py) now reads confidence with read_confidence and the same rule. It had the ×100 sizing bug: 25 bought 500 shares.
  • While testing that, I found that the loop's LLM path has raised ImportError on every decision since chore: sweep CodeQL quality alerts #216 removed the timedelta re-export it imported from backtest_hourly_agent. It now imports timedelta directly.
  • The AI Hedge Fund adapter divides its 0–100 scale out and asks the shared confidence_in_range. A huge int no longer raises OverflowError past the check.

Docs. SECURITY.md's constraint table gets an "Unreadable confidence" row. The "preserved byte-for-byte" claim is replaced with the versioned-gate rule.

Tests. The full backend suite passes locally: 7242 passed, 0 failed. New tests are in test_portfolio_manager_move.py (sell / sized / unsized / strict / counts across 25, 80, 1.5, -0.5, NaN, Infinity, true, "high"), test_deploy_guard.py, test_market_data_provenance.py, test_agent_runs_metadata.py, domain/agents/test_runtime.py and test_custom_algo_confidence.py.

Merged with #662 and #667 (0321ac6). Nine files conflicted. Each side's change is kept, plus one fix: #667's sell branch formatted {confidence:.0%}, which raises on the unreadable confidence this gate lets a sell trade on, so it now prints unread. Runs record both llm_gate_version and llm_sell_sizing, and the live board's resume key is unchanged (it still leaves out the gate version). The duplicate buy-size helper is folded into main's _coerce_position_size. The Trading Algo ImportError was also fixed in #662, and the two fixes merged identically. Full backend suite after the merge: 7358 passed, 0 failed.

🤖 Generated with Claude Code

make_trading_decision_with_llm read confidence with no range check. A model answering on a 0-100 scale (25 meaning 25%) cleared the 0.3 floor and multiplied the confidence-sized buy by 100: confidence 25 bought 500 shares (50% of a 100k account). NaN and Infinity traded too. Such an action is now skipped and logged, the same 0-1 rule validate_confidence applies to external agents; no clamping or rescaling.
@vercel

vercel Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel.

A member of the Team first needs to authorize it.

@FlyM1ss

FlyM1ss commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

#673 is merged and applies the same 0–1 rule at the gate (read_confidence in validator.py), so this skip is now duplicated, and the branch conflicts with main. Checked against #673's branch: 25, 80, 1.5, -0.5, NaN and Infinity skip, and 0.3, 1.0 and 1 trade, as your tests expect. Only the "outside 0-1" log asserts fail, since the message on main reads is not a number from 0 to 1. Suggest closing this, or keeping just the tests with that string updated.

FlyM1ss and others added 4 commits October 30, 2026 00:44
…ing the order

main skips every action whose confidence is not a number from 0 to 1. Only
an unsized buy is sized from it, so that rule also dropped sells (a model
on a 0-100 scale could never exit) and orders with their own size (pipeline
prompts name no scale). Such a confidence is now not read at all: it
neither sizes a buy nor decides the 0.3 floor. Sells and sized buys trade
as written; an unsized buy is skipped, or rejects the batch on a strict
run like a bad symbol.

A run that breaks the rule is no longer a quiet flat curve: one flushed
ERROR: llm.unusable_confidence line, counts in agent_runs.metadata, and the
leaderboard's H6 guard takes the affected steps off its numerator.
llm_decisions is left alone because the decision tape reads an unchanged
count as a fallback bar.

Every LLM run records llm_gate_version. It joins the sample pooling key, and
cached rows from another gate print a warning (no auto-refresh: that would
be a billable deploy from a public GET). SECURITY.md's table is updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cate

The adapter's 0-100 check was a third copy of the confidence range rule.
It now divides its scale out and asks validator.confidence_in_range, and
reads the number with model_number, so an int too large for float() is out
of range instead of raising OverflowError past the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Trading Algo loop read confidence with float() and no range check, so
25 cleared the 0.3 floor and multiplied the auto-sized buy by 100 (500
shares on $100k). It now uses read_confidence with the gate's rule.

Its LLM path also raised ImportError on every decision: it imported
timedelta from backtest_hourly_agent, whose re-export Open-Finance-Lab#216 removed as
unused. timedelta is now imported directly; the new test drives that path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss FlyM1ss changed the title fix(backtests): skip an LLM action whose confidence is outside 0-1 fix(backtests): stop reading LLM confidence off the 0-1 scale Oct 30, 2026
…confidence-range

Conflicts resolved:
- portfolio_manager.py: kept main's `_coerce_position_size` and dropped this
  branch's identical `_requested_buy_shares`; kept both `recognised_any` (Open-Finance-Lab#658)
  and `skipped_for_confidence`. Open-Finance-Lab#667's sell branch formatted
  `{confidence:.0%}`, which raises on the unreadable confidence this gate lets
  a sell trade on; it now uses `conf_label`.
- engine.py, leaderboard/service.py and their metadata tests: record both
  `llm_gate_version` and `llm_sell_sizing`.
- validator.py: Open-Finance-Lab#667's sell-sizing block, then the public
  `confidence_in_range`.
- backtest_custom_algo.py, ai_hedge_fund/adapter.py: Open-Finance-Lab#662's
  normalize_symbol/normalize_action alongside read_confidence and
  confidence_in_range.
- SECURITY.md: the versioned-gate paragraph now names `llm_sell_sizing` and
  the reply readings; the Trading Algo row lists all three shared readers.

LLM_GATE_VERSION's history notes that v2, the first recorded version, also
covers Open-Finance-Lab#658 and Open-Finance-Lab#661.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss
FlyM1ss merged commit c958f53 into Open-Finance-Lab:main Oct 30, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants