Skip to content

fix(backtests): read every LLM action's confidence by one 0-1 rule - #673

Merged
FlyM1ss merged 4 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/pipeline-zero-confidence
Oct 30, 2026
Merged

FlyM1ss merged 4 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/pipeline-zero-confidence

Conversation

@subidhkhanal

@subidhkhanal subidhkhanal commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

The bug

pipeline_output_to_decision (infrastructure/llm/pipeline_runner.py, orders branch) read an order's confidence as:

"confidence": float(order.get("confidence", 0.75) or 0.75),

The or 0.75 was meant as a default for a missing confidence. In Python, 0 or 0.75 is 0.75, though, because 0 is falsy. So a pipeline order with confidence: 0 ("not confident at all") was read as 0.75, passed the 0.3 floor in make_trading_decision_with_llm and traded.

Reproduced on main before the fix. Running a pipeline reply of {"orders": [{"symbol": "AAPL", "side": "buy", "qty": 10, "confidence": 0}]} through PortfolioManager.make_trading_decision_with_llm placed the buy:

✅ BUY AAPL: 10 shares @ $100.00 (conf: 75%)

The same "not confident, but it still trades" shape, and its crash variants, also sat in nearby code. A review of the first fix found them:

Model wrote Before
confidence: [] / {} / "high" raised out of the converter: the step's usage was lost and a paid run aborted after its calls were charged
confidence: NaN / Infinity / 20 cleared the 0.3 floor (nan < 0.3 is False) and traded
risk_actions confidence: 0 hard-coded to 0.8 and sold
actions reply confidence: "0.9" / null reached {confidence:.0%} raw: ValueError / TypeError, a paid run aborted
qty: "10.5" / NaN / 0.6 int() turned it into 0, which the gate reads as "size it for me"
qty: Infinity OverflowError, a paid run aborted
size_pct: 0 / "50%" 0 read as 1.0, the whole position; "50%" raised

The fix

One rule, read_confidence in infrastructure/llm/validator.py, used by the gate and by every pipeline envelope it reads:

  • Left out (missing, null, ""): the path's default.
  • A number from 0 to 1, or a numeric string ("0.4"): kept, 0 included.
  • Anything else the model gave (a bool, "high", a list, NaN, infinity, 25): the action is skipped and logged. No guessing a scale or a default over it. This is the same 0–1 range LLMTradingDecision enforces for external agents.

Applied:

  • orders and risk_actions confidence: read by this rule. An unusable value is handed on as written, so the gate skips the action and logs what the model sent.
  • The gate: reads every action through it, so an actions reply gets the same treatment.
  • qty: handed on as written. The gate already coerces it, skips a size it cannot read and sizes a missing or zero one from confidence.
  • size_pct: missing still sells the whole position. A size the model gave must be a positive number, otherwise the action holds.

Behaviour changes worth knowing:

  • confidence: null on an actions reply now trades at the 0.5 default. It used to crash the gate.
  • true / false are skipped. Before, false read as 0.75 on the pipeline path and true as 1.0 on the gate.
  • qty: 10.7 now reaches the gate as 10.7, not the converter's 10. On a lot-traded market that means an off-lot rejection instead of a silent correction.

Defaults are named but not unified: 0.5 for an actions reply, 0.75 for pipeline orders, 0.8 for risk_actions. The default instruction asks for no confidence, so its orders carry 0.75 and one without a qty is sized from it. Moving to 0.5 would shrink those buys by a third and break comparability with earlier runs.

Overlap with open work

Tests

  • test_llm_validator.py::TestReadConfidence: the rule itself.
  • test_pipeline_runner.py: every row of the table at the converter (confidence, qty, risk_actions confidence and size_pct).
  • test_portfolio_manager_move.py: end to end on a fail-closed (billed) client. Each unusable confidence and unreadable qty is one skipped order on a bar the model still drove (llm_calls == 1, llm_decisions == 1), not an aborted run. actions replies with "0.9"/null/"0.2" trade or skip as expected.
  • test_decision_tape_engine.py: the replay test now expects confidence: null to trade at the default rather than fall back.
  • test_backtest_harness.py::test_post_parse_exception_is_billed_but_not_a_decision used confidence: "0.9" to make the processing loop raise. That is a usable confidence now, so the test raises with action: 7 instead and still pins the same H6 rule (billed, not a decision).
  • 37 of the new cases fail on this PR's first commit and pass now. Full backend suite: 7179 passed, 0 failed (local; test_report_pdf.py and test_robinhood_mcp_transport.py excluded, both environmental here and green in CI).

🤖 Generated with Claude Code

…ading it as 0.75

pipeline_output_to_decision read an order's confidence as float(value or 0.75), so confidence 0 became 0.75, passed the 0.3 floor and traded. Only a missing or empty confidence now takes the 0.75 default.
@vercel

vercel Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel.

A member of the Team first needs to authorize it.

FlyM1ss and others added 3 commits October 29, 2026 22:38
Review of the zero-confidence fix found the same "not confident, but it
still trades" shape, and crash variants, around it:

- [] / {} / "high" confidence raised out of pipeline_output_to_decision,
  losing the step's usage and aborting a paid run after its calls were
  charged; NaN, Infinity and 20 cleared the 0.3 floor (nan < 0.3 is False).
- risk_actions hard-coded confidence 0.8, so a zero-confidence stop-loss
  sold; size_pct 0 read as 1.0 (the whole position), "50%" raised.
- an `actions` reply reached the gate's {confidence:.0%} raw, so "0.9"
  and null raised.
- int(qty) turned "10.5", NaN and 0.6 into 0 (auto-size) and raised
  OverflowError on Infinity.

read_confidence (validator.py) is now the one rule for the gate and every
pipeline envelope: a missing confidence takes the path's default, a number
from 0 to 1 (or a numeric string) is kept, anything else skips the action.
The converter hands unusable confidences and every qty on as written, so the
gate skips and logs what the model sent. The three defaults (0.5, 0.75,
0.8) are named in one place but kept apart: unifying would resize every
default-instruction pipeline order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A numeric-string confidence ("0.9") is now read by the 0-1 rule instead of
crashing the processing loop, so the H6 test that relied on that crash raises
through action.lower() instead. It still pins the same rule: the call is
billed but is not counted as a decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss FlyM1ss changed the title fix(backtests): keep a pipeline order's zero confidence instead of reading it as 0.75 fix(backtests): read every LLM action's confidence by one 0-1 rule Oct 30, 2026
@FlyM1ss
FlyM1ss merged commit bc695c0 into Open-Finance-Lab:main Oct 30, 2026
8 checks passed
FlyM1ss added a commit to subidhkhanal/AgenticTrading that referenced this pull request Oct 30, 2026
Open-Finance-Lab#658 was two readings of one field: the strict check stripped the action,
the trade loop did not, so "buy " passed one and matched nothing in the
other. This puts every reader of a model's action on one helper and makes
an action nobody recognises visible instead of a silent hold.

- validator: normalize_action (ASCII trim + ASCII lowercase, None -> "",
  non-strings via str) and TRADING_ACTIONS, beside normalize_symbol.
- portfolio_manager: the strict check and the trade loop both use it. A
  non-strict loop skips an unrecognised action with a log line instead of
  treating it as a hold, and a batch with no recognised action holds
  without crediting llm_decisions (H6). A null or non-text reasoning no
  longer raises (None[:60] aborted billed runs and sent lenient ones to
  the rule-based fallback); neither does a null or numeric action.
- pipeline_runner: a blank side falls through to the action alias;
  unrecognised sides and risk actions are handed on as written for the
  gate to refuse (as Open-Finance-Lab#673 did for size), instead of being coerced to hold.
- decision_tape: _order_triples reads action and symbol as the gate does,
  so gate_rewrote stops flagging a padded order the gate accepted.
- AI Hedge Fund adapter, Robinhood live service, Trading Algo backtest:
  same helper for the action (and the symbol, in the Trading Algo).
- backtest_custom_algo: import timedelta from datetime. cd402c8 removed
  it from backtest_hourly_agent, so every Trading Algo run raised
  ImportError on its first decision.
- SECURITY.md: action vocabulary row; the replay note no longer promises
  byte-for-byte reproduction across gate changes.
- tests: fixture keys fixed (sma20/sma50); strict refusal, lenient skip,
  H6 credit, non-text fields, pipeline sides, the helper itself, the
  Robinhood path, and a Trading Algo subprocess run. Two tests that pinned
  the old crashes now pin the new behaviour; the harness's blanket-except
  test injects its fault instead of relying on a malformed field.

Behaviour change: a pipeline side such as "Buy." used to hold silently;
a strict (billed) run now refuses it and aborts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlyM1ss added a commit to subidhkhanal/AgenticTrading that referenced this pull request Oct 30, 2026
One conflict, in pipeline_output_to_decision's risk branch. Kept main's
reading of size_pct (Open-Finance-Lab#673: a missing size sells all, an unusable or
non-positive one holds) and confidence; a SELL now carries size_pct as
size_fraction with position_size 0, as this branch intends, instead of
main's round(size_pct * 100) share count. The risk test main added is
updated to the fraction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlyM1ss added a commit to subidhkhanal/AgenticTrading that referenced this pull request Oct 30, 2026
…rything

Review follow-ups to the sell-sizing change (Open-Finance-Lab#661), on top of main (Open-Finance-Lab#673):

- One sizing rule, validator.requested_sell_shares, for the gate and for
  actions_to_executable. Only a missing, null or 0 size means the whole
  position. A negative, fractional, non-numeric, bool or non-finite
  position_size or size_fraction is refused rather than read as "sell all"
  (-0.5 and false used to liquidate). A trim that rounds to no shares is
  refused too. The sign is read before anything truncates, a 400-digit
  size_fraction no longer raises OverflowError, and 0.57 x 100 sells 57,
  not 56.
- A refused SELL is recorded as an invalid_quantity rejection on
  order_events and rejected_orders, collapsed like an executor rejection.
  It used to be only a print in a subprocess. Conformance C04/DL's o2
  (sell -5) now matches the reference, and the baseline is re-recorded.
- Prompts: SAFE_TRADING_PROMPT and the custom-strategy contract said
  "otherwise use 1" and showed position_size 1. They now say a SELL's
  position_size is the shares to sell (0 = all). The starter instruction
  (defaults.py, app.js, 7 marketplace cards) says to set qty to all the
  shares held. The pipeline's EXECUTION RULES state the rule on every run,
  for agents seeded with the old sentence. The old wording still counts as
  the default (PRIOR_STARTER_INSTRUCTIONS). app.js v=163.
- Runs record llm_sell_sizing. A leaderboard pool never mixes rows of two
  rules (_SAMPLE_CONFIG_KEYS). A Live checkpoint without the key is read
  as the old rule, so its month is replayed, not spliced.
- Decision tape v2. gate_rewrote pairs sizes as a multiset per symbol and
  side, so a split sell is no longer called a rewrite.
- SECURITY.md table, conformance inventory and the tape and conformance
  design docs updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants