Repository navigation
fix(backtests): read every LLM action's confidence by one 0-1 rule - #673
Merged
FlyM1ss merged 4 commits intoOct 30, 2026
Merged
Conversation
…ading it as 0.75 pipeline_output_to_decision read an order's confidence as float(value or 0.75), so confidence 0 became 0.75, passed the 0.3 floor and traded. Only a missing or empty confidence now takes the 0.75 default.
Contributor
|
@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel. A member of the Team first needs to authorize it. |
Review of the zero-confidence fix found the same "not confident, but it
still trades" shape, and crash variants, around it:
- [] / {} / "high" confidence raised out of pipeline_output_to_decision,
losing the step's usage and aborting a paid run after its calls were
charged; NaN, Infinity and 20 cleared the 0.3 floor (nan < 0.3 is False).
- risk_actions hard-coded confidence 0.8, so a zero-confidence stop-loss
sold; size_pct 0 read as 1.0 (the whole position), "50%" raised.
- an `actions` reply reached the gate's {confidence:.0%} raw, so "0.9"
and null raised.
- int(qty) turned "10.5", NaN and 0.6 into 0 (auto-size) and raised
OverflowError on Infinity.
read_confidence (validator.py) is now the one rule for the gate and every
pipeline envelope: a missing confidence takes the path's default, a number
from 0 to 1 (or a numeric string) is kept, anything else skips the action.
The converter hands unusable confidences and every qty on as written, so the
gate skips and logs what the model sent. The three defaults (0.5, 0.75,
0.8) are named in one place but kept apart: unifying would resize every
default-instruction pipeline order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A numeric-string confidence ("0.9") is now read by the 0-1 rule instead of
crashing the processing loop, so the H6 test that relied on that crash raises
through action.lower() instead. It still pins the same rule: the call is
billed but is not counted as a decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 30, 2026
FlyM1ss
added a commit
to subidhkhanal/AgenticTrading
that referenced
this pull request
Oct 30, 2026
Open-Finance-Lab#658 was two readings of one field: the strict check stripped the action, the trade loop did not, so "buy " passed one and matched nothing in the other. This puts every reader of a model's action on one helper and makes an action nobody recognises visible instead of a silent hold. - validator: normalize_action (ASCII trim + ASCII lowercase, None -> "", non-strings via str) and TRADING_ACTIONS, beside normalize_symbol. - portfolio_manager: the strict check and the trade loop both use it. A non-strict loop skips an unrecognised action with a log line instead of treating it as a hold, and a batch with no recognised action holds without crediting llm_decisions (H6). A null or non-text reasoning no longer raises (None[:60] aborted billed runs and sent lenient ones to the rule-based fallback); neither does a null or numeric action. - pipeline_runner: a blank side falls through to the action alias; unrecognised sides and risk actions are handed on as written for the gate to refuse (as Open-Finance-Lab#673 did for size), instead of being coerced to hold. - decision_tape: _order_triples reads action and symbol as the gate does, so gate_rewrote stops flagging a padded order the gate accepted. - AI Hedge Fund adapter, Robinhood live service, Trading Algo backtest: same helper for the action (and the symbol, in the Trading Algo). - backtest_custom_algo: import timedelta from datetime. cd402c8 removed it from backtest_hourly_agent, so every Trading Algo run raised ImportError on its first decision. - SECURITY.md: action vocabulary row; the replay note no longer promises byte-for-byte reproduction across gate changes. - tests: fixture keys fixed (sma20/sma50); strict refusal, lenient skip, H6 credit, non-text fields, pipeline sides, the helper itself, the Robinhood path, and a Trading Algo subprocess run. Two tests that pinned the old crashes now pin the new behaviour; the harness's blanket-except test injects its fault instead of relying on a malformed field. Behaviour change: a pipeline side such as "Buy." used to hold silently; a strict (billed) run now refuses it and aborts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlyM1ss
added a commit
to subidhkhanal/AgenticTrading
that referenced
this pull request
Oct 30, 2026
One conflict, in pipeline_output_to_decision's risk branch. Kept main's reading of size_pct (Open-Finance-Lab#673: a missing size sells all, an unusable or non-positive one holds) and confidence; a SELL now carries size_pct as size_fraction with position_size 0, as this branch intends, instead of main's round(size_pct * 100) share count. The risk test main added is updated to the fraction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlyM1ss
added a commit
to subidhkhanal/AgenticTrading
that referenced
this pull request
Oct 30, 2026
…rything Review follow-ups to the sell-sizing change (Open-Finance-Lab#661), on top of main (Open-Finance-Lab#673): - One sizing rule, validator.requested_sell_shares, for the gate and for actions_to_executable. Only a missing, null or 0 size means the whole position. A negative, fractional, non-numeric, bool or non-finite position_size or size_fraction is refused rather than read as "sell all" (-0.5 and false used to liquidate). A trim that rounds to no shares is refused too. The sign is read before anything truncates, a 400-digit size_fraction no longer raises OverflowError, and 0.57 x 100 sells 57, not 56. - A refused SELL is recorded as an invalid_quantity rejection on order_events and rejected_orders, collapsed like an executor rejection. It used to be only a print in a subprocess. Conformance C04/DL's o2 (sell -5) now matches the reference, and the baseline is re-recorded. - Prompts: SAFE_TRADING_PROMPT and the custom-strategy contract said "otherwise use 1" and showed position_size 1. They now say a SELL's position_size is the shares to sell (0 = all). The starter instruction (defaults.py, app.js, 7 marketplace cards) says to set qty to all the shares held. The pipeline's EXECUTION RULES state the rule on every run, for agents seeded with the old sentence. The old wording still counts as the default (PRIOR_STARTER_INSTRUCTIONS). app.js v=163. - Runs record llm_sell_sizing. A leaderboard pool never mixes rows of two rules (_SAMPLE_CONFIG_KEYS). A Live checkpoint without the key is read as the old rule, so its month is replayed, not spliced. - Decision tape v2. gate_rewrote pairs sizes as a multiset per symbol and side, so a split sell is no longer called a rewrite. - SECURITY.md table, conformance inventory and the tape and conformance design docs updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
pipeline_output_to_decision(infrastructure/llm/pipeline_runner.py,ordersbranch) read an order's confidence as:The
or 0.75was meant as a default for a missing confidence. In Python,0 or 0.75is0.75, though, because 0 is falsy. So a pipeline order withconfidence: 0("not confident at all") was read as 0.75, passed the 0.3 floor inmake_trading_decision_with_llmand traded.Reproduced on
mainbefore the fix. Running a pipeline reply of{"orders": [{"symbol": "AAPL", "side": "buy", "qty": 10, "confidence": 0}]}throughPortfolioManager.make_trading_decision_with_llmplaced the buy:The same "not confident, but it still trades" shape, and its crash variants, also sat in nearby code. A review of the first fix found them:
confidence: []/{}/"high"confidence: NaN/Infinity/20nan < 0.3is False) and tradedrisk_actionsconfidence: 0actionsreplyconfidence: "0.9"/null{confidence:.0%}raw: ValueError / TypeError, a paid run abortedqty: "10.5"/NaN/0.6int()turned it into 0, which the gate reads as "size it for me"qty: InfinityOverflowError, a paid run abortedsize_pct: 0/"50%""50%"raisedThe fix
One rule,
read_confidenceininfrastructure/llm/validator.py, used by the gate and by every pipeline envelope it reads:null,""): the path's default."0.4"): kept,0included."high", a list, NaN, infinity,25): the action is skipped and logged. No guessing a scale or a default over it. This is the same 0–1 rangeLLMTradingDecisionenforces for external agents.Applied:
ordersandrisk_actionsconfidence: read by this rule. An unusable value is handed on as written, so the gate skips the action and logs what the model sent.actionsreply gets the same treatment.qty: handed on as written. The gate already coerces it, skips a size it cannot read and sizes a missing or zero one from confidence.size_pct: missing still sells the whole position. A size the model gave must be a positive number, otherwise the action holds.Behaviour changes worth knowing:
confidence: nullon anactionsreply now trades at the 0.5 default. It used to crash the gate.true/falseare skipped. Before,falseread as 0.75 on the pipeline path andtrueas 1.0 on the gate.qty: 10.7now reaches the gate as 10.7, not the converter's 10. On a lot-traded market that means an off-lot rejection instead of a silent correction.Defaults are named but not unified: 0.5 for an
actionsreply, 0.75 for pipelineorders, 0.8 forrisk_actions. The default instruction asks for no confidence, so its orders carry 0.75 and one without a qty is sized from it. Moving to 0.5 would shrink those buys by a third and break comparability with earlier runs.Overlap with open work
risk_actionsnormalisation and the gate's buy sizing. Whichever lands second has a small conflict here.Tests
test_llm_validator.py::TestReadConfidence: the rule itself.test_pipeline_runner.py: every row of the table at the converter (confidence, qty,risk_actionsconfidence andsize_pct).test_portfolio_manager_move.py: end to end on a fail-closed (billed) client. Each unusable confidence and unreadable qty is one skipped order on a bar the model still drove (llm_calls == 1,llm_decisions == 1), not an aborted run.actionsreplies with"0.9"/null/"0.2"trade or skip as expected.test_decision_tape_engine.py: the replay test now expectsconfidence: nullto trade at the default rather than fall back.test_backtest_harness.py::test_post_parse_exception_is_billed_but_not_a_decisionusedconfidence: "0.9"to make the processing loop raise. That is a usable confidence now, so the test raises withaction: 7instead and still pins the same H6 rule (billed, not a decision).test_report_pdf.pyandtest_robinhood_mcp_transport.pyexcluded, both environmental here and green in CI).🤖 Generated with Claude Code