Repository navigation
fix(backtests): stop reading LLM confidence off the 0-1 scale - #674
Merged
FlyM1ss merged 6 commits intoOct 30, 2026
Merged
Conversation
make_trading_decision_with_llm read confidence with no range check. A model answering on a 0-100 scale (25 meaning 25%) cleared the 0.3 floor and multiplied the confidence-sized buy by 100: confidence 25 bought 500 shares (50% of a 100k account). NaN and Infinity traded too. Such an action is now skipped and logged, the same 0-1 rule validate_confidence applies to external agents; no clamping or rescaling.
Contributor
|
@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel. A member of the Team first needs to authorize it. |
Collaborator
|
#673 is merged and applies the same 0–1 rule at the gate ( |
…ing the order main skips every action whose confidence is not a number from 0 to 1. Only an unsized buy is sized from it, so that rule also dropped sells (a model on a 0-100 scale could never exit) and orders with their own size (pipeline prompts name no scale). Such a confidence is now not read at all: it neither sizes a buy nor decides the 0.3 floor. Sells and sized buys trade as written; an unsized buy is skipped, or rejects the batch on a strict run like a bad symbol. A run that breaks the rule is no longer a quiet flat curve: one flushed ERROR: llm.unusable_confidence line, counts in agent_runs.metadata, and the leaderboard's H6 guard takes the affected steps off its numerator. llm_decisions is left alone because the decision tape reads an unchanged count as a fallback bar. Every LLM run records llm_gate_version. It joins the sample pooling key, and cached rows from another gate print a warning (no auto-refresh: that would be a billable deploy from a public GET). SECURITY.md's table is updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cate The adapter's 0-100 check was a third copy of the confidence range rule. It now divides its scale out and asks validator.confidence_in_range, and reads the number with model_number, so an int too large for float() is out of range instead of raising OverflowError past the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Trading Algo loop read confidence with float() and no range check, so 25 cleared the 0.3 floor and multiplied the auto-sized buy by 100 (500 shares on $100k). It now uses read_confidence with the gate's rule. Its LLM path also raised ImportError on every decision: it imported timedelta from backtest_hourly_agent, whose re-export Open-Finance-Lab#216 removed as unused. timedelta is now imported directly; the new test drives that path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…confidence-range Conflicts resolved: - portfolio_manager.py: kept main's `_coerce_position_size` and dropped this branch's identical `_requested_buy_shares`; kept both `recognised_any` (Open-Finance-Lab#658) and `skipped_for_confidence`. Open-Finance-Lab#667's sell branch formatted `{confidence:.0%}`, which raises on the unreadable confidence this gate lets a sell trade on; it now uses `conf_label`. - engine.py, leaderboard/service.py and their metadata tests: record both `llm_gate_version` and `llm_sell_sizing`. - validator.py: Open-Finance-Lab#667's sell-sizing block, then the public `confidence_in_range`. - backtest_custom_algo.py, ai_hedge_fund/adapter.py: Open-Finance-Lab#662's normalize_symbol/normalize_action alongside read_confidence and confidence_in_range. - SECURITY.md: the versioned-gate paragraph now names `llm_sell_sizing` and the reply readings; the Trading Algo row lists all three shared readers. LLM_GATE_VERSION's history notes that v2, the first recorded version, also covers Open-Finance-Lab#658 and Open-Finance-Lab#661. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
make_trading_decision_with_llm(domain/backtesting/portfolio_manager.py) reads each action'sconfidencewith no range check. Confidence is meant to be 0–1. A model that answers on a 0–100 scale (25meaning 25%):25 < 0.3is false), although 25% is below it;position_sizeis given (risk_amount = 2% of equity × confidence).Reproduced on
mainwith{"action": "buy", "symbol": "AAPL", "confidence": 25}on a $100k account:That is $50k, half the account. A correct 0.25 would have been skipped by the floor. Other values on
main:1.5bought at "150%";NaNandInfinityboth traded;80was dropped only because the inflated buy needed $160k of cash.The fix
Before the 0.3 floor, an action whose confidence is a number outside
[0, 1], or is not finite, is skipped and logged:validate_confidenceininfrastructure/llm/validator.pyalready enforces for external agents.Unchanged:
0.3and1.0still trade;"high",null) behaves as before, which is fix(backtests): a pipeline reply with a non-numeric confidence or qty aborts a paid run and loses the step's usage #663's scope;Tests
New cases in
tests/backtesting/test_portfolio_manager_move.py:25,80and1.5→ no action, plus the skip line;-0.5,NaNandInfinity→ no action, plus the skip line;0.3,1.0and1→ the buy still goes through.The out-of-range cases failed on
mainas listed above and pass with the fix. Focused suites (tests/backtesting,tests/domain/backtesting,tests/infrastructure/llm,tests/conformance): 1140 passed, 0 failed.Maintainer follow-up
The sections above describe the branch's original commit. The title changed from "skip an LLM action whose confidence is outside 0-1" because the branch no longer skips every such action.
#673 merged first and already skips any action whose confidence is outside 0–1 (
read_confidence), so this branch's own hunk was a duplicate. The merge ofmainkeeps #673's version. The commits after it narrow that rule to where confidence actually drives the outcome, and make a run that breaks it visible.Gate (
LLM_GATE_VERSION = 2). A confidence that is not a number from 0 to 1 (25, NaN,"high", a bool) is not read at all. It neither sizes a buy nor decides the 0.3 floor.position_sizetrades as written. Onmainthese were dropped, so a model on a 0–100 scale could never exit a position. Pipeline prompts name no scale.Visible, not a flat curve.
ERROR: llm.unusable_confidenceline per run, which the parent relays to the service log.agent_runs.metadata.unusable_confidence, from the engine and the leaderboard. The live board adds them across checkpoints.confidence_skipped_stepsoff its numerator.llm_decisionsis untouched, because the decision tape reads an unchanged count as a fallback bar.Provenance.
llm_gate_versionis recorded on every LLM run.Same bug elsewhere.
backtest_custom_algo.py) now reads confidence withread_confidenceand the same rule. It had the ×100 sizing bug:25bought 500 shares.ImportErroron every decision since chore: sweep CodeQL quality alerts #216 removed thetimedeltare-export it imported frombacktest_hourly_agent. It now importstimedeltadirectly.confidence_in_range. A huge int no longer raisesOverflowErrorpast the check.Docs.
SECURITY.md's constraint table gets an "Unreadable confidence" row. The "preserved byte-for-byte" claim is replaced with the versioned-gate rule.Tests. The full backend suite passes locally: 7242 passed, 0 failed. New tests are in
test_portfolio_manager_move.py(sell / sized / unsized / strict / counts across25,80,1.5,-0.5, NaN, Infinity,true,"high"),test_deploy_guard.py,test_market_data_provenance.py,test_agent_runs_metadata.py,domain/agents/test_runtime.pyandtest_custom_algo_confidence.py.Merged with #662 and #667 (0321ac6). Nine files conflicted. Each side's change is kept, plus one fix: #667's sell branch formatted
{confidence:.0%}, which raises on the unreadable confidence this gate lets a sell trade on, so it now printsunread. Runs record bothllm_gate_versionandllm_sell_sizing, and the live board's resume key is unchanged (it still leaves out the gate version). The duplicate buy-size helper is folded into main's_coerce_position_size. The Trading AlgoImportErrorwas also fixed in #662, and the two fixes merged identically. Full backend suite after the merge: 7358 passed, 0 failed.🤖 Generated with Claude Code