Skip to content

fix(backtests): strip the model's action before matching it - #662

Merged
FlyM1ss merged 3 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-action-whitespace
Oct 30, 2026
Merged

FlyM1ss merged 3 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-action-whitespace

Conversation

@subidhkhanal

@subidhkhanal subidhkhanal commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Closes #658

The problem

If the model writes its action with an extra space, like "buy ", the trade is silently skipped, but the step still counts as a model decision.

The safety check removes the space, so "buy " passes. The code that places the trade doesn't, so "buy " doesn't match "buy" and nothing happens. The pipeline path had the same problem: "buy " was turned into "hold".

Model writes Before After
"buy" Buy placed Buy placed
"buy " Nothing placed, still counted Buy placed

The fix

Remove the extra spaces before comparing, the same way the safety check already does. That's a .strip() in three places:

  • portfolio_manager.py: the loop that places trades
  • pipeline_runner.py: pipeline orders and risk_actions

Tests

  • "buy " and " BUY\n" now place one buy, in strict and normal mode.
  • A pipeline "buy " stays "buy", and " stop_loss\n" becomes "sell".
  • The new tests fail on main and pass here.

Note

robinhood_live_service.py has the same pattern, but that real-money path was switched off in #656, so I left it alone.

Maintainer follow-up (f34a27e, 0639336)

Merged main in (conflicts with normalize_symbol / read_confidence), then fixed what the review found:

  • One rule for the action: validator.normalize_action (ASCII trim + lowercase) is used by the strict check, the trade loop, the pipeline converter, the decision tape, the AI Hedge Fund adapter, the Trading Algo, and the Robinhood path (so the note above is now handled).
  • Unrecognised actions are visible: a lenient run skips them with a log line; a batch where nothing was buy/sell/hold holds the step without H6 credit. The pipeline no longer coerces an unknown side to hold; it hands it on and the gate decides, as fix(backtests): read every LLM action's confidence by one 0-1 rule #673 did for size.
  • Crashes on the same fields: a null or non-text reasoning, and a null or numeric action, no longer throw the step into the fallback (a billed run used to abort on None[:60]).
  • Trading Algo was down: backtest_custom_algo.py imported timedelta from backtest_hourly_agent, which cd402c8a removed, so every run raised ImportError on its first decision. Fixed, with a subprocess test.
  • SECURITY.md: action-vocabulary row; replays no longer promise byte-for-byte reproduction across gate changes.

Behaviour change: a pipeline side like "Buy." used to hold silently; a strict (billed) run now refuses it and aborts.

Full backend suite green apart from the known local-only test_report_pdf (reportlab) and test_robinhood_mcp_transport (mcp version) failures.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel.

A member of the Team first needs to authorize it.

FlyM1ss and others added 2 commits October 30, 2026 00:44
Open-Finance-Lab#658 was two readings of one field: the strict check stripped the action,
the trade loop did not, so "buy " passed one and matched nothing in the
other. This puts every reader of a model's action on one helper and makes
an action nobody recognises visible instead of a silent hold.

- validator: normalize_action (ASCII trim + ASCII lowercase, None -> "",
  non-strings via str) and TRADING_ACTIONS, beside normalize_symbol.
- portfolio_manager: the strict check and the trade loop both use it. A
  non-strict loop skips an unrecognised action with a log line instead of
  treating it as a hold, and a batch with no recognised action holds
  without crediting llm_decisions (H6). A null or non-text reasoning no
  longer raises (None[:60] aborted billed runs and sent lenient ones to
  the rule-based fallback); neither does a null or numeric action.
- pipeline_runner: a blank side falls through to the action alias;
  unrecognised sides and risk actions are handed on as written for the
  gate to refuse (as Open-Finance-Lab#673 did for size), instead of being coerced to hold.
- decision_tape: _order_triples reads action and symbol as the gate does,
  so gate_rewrote stops flagging a padded order the gate accepted.
- AI Hedge Fund adapter, Robinhood live service, Trading Algo backtest:
  same helper for the action (and the symbol, in the Trading Algo).
- backtest_custom_algo: import timedelta from datetime. cd402c8 removed
  it from backtest_hourly_agent, so every Trading Algo run raised
  ImportError on its first decision.
- SECURITY.md: action vocabulary row; the replay note no longer promises
  byte-for-byte reproduction across gate changes.
- tests: fixture keys fixed (sma20/sma50); strict refusal, lenient skip,
  H6 credit, non-text fields, pipeline sides, the helper itself, the
  Robinhood path, and a Trading Algo subprocess run. Two tests that pinned
  the old crashes now pin the new behaviour; the harness's blanket-except
  test injects its fault instead of relying on a malformed field.

Behaviour change: a pipeline side such as "Buy." used to hold silently;
a strict (billed) run now refuses it and aborts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss
FlyM1ss merged commit 7d83f89 into Open-Finance-Lab:main Oct 30, 2026
9 checks passed
FlyM1ss added a commit to subidhkhanal/AgenticTrading that referenced this pull request Oct 30, 2026
Conflicts with Open-Finance-Lab#662 (normalize_action), all combined:
- SECURITY.md: main's count-cap Action count row, this branch's split
  BUY/SELL per-order size rows.
- decision_tape._order_triples: main's normalize_symbol, this branch's
  sell sizes and unsorted triples (gate_rewrote pairs them as a multiset).
- portfolio_manager imports: both model_number and normalize_action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FlyM1ss added a commit to subidhkhanal/AgenticTrading that referenced this pull request Oct 30, 2026
…confidence-range

Conflicts resolved:
- portfolio_manager.py: kept main's `_coerce_position_size` and dropped this
  branch's identical `_requested_buy_shares`; kept both `recognised_any` (Open-Finance-Lab#658)
  and `skipped_for_confidence`. Open-Finance-Lab#667's sell branch formatted
  `{confidence:.0%}`, which raises on the unreadable confidence this gate lets
  a sell trade on; it now uses `conf_label`.
- engine.py, leaderboard/service.py and their metadata tests: record both
  `llm_gate_version` and `llm_sell_sizing`.
- validator.py: Open-Finance-Lab#667's sell-sizing block, then the public
  `confidence_in_range`.
- backtest_custom_algo.py, ai_hedge_fund/adapter.py: Open-Finance-Lab#662's
  normalize_symbol/normalize_action alongside read_confidence and
  confidence_in_range.
- SECURITY.md: the versioned-gate paragraph now names `llm_sell_sizing` and
  the reply readings; the Trading Algo row lists all three shared readers.

LLM_GATE_VERSION's history notes that v2, the first recorded version, also
covers Open-Finance-Lab#658 and Open-Finance-Lab#661.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
computedocean pushed a commit to computedocean/AgenticTrading that referenced this pull request Oct 31, 2026
The risk-branch comment in pipeline_output_to_decision said any
positive size_pct sells the whole position because the gate does not
size a sell. That was written against the pre-Open-Finance-Lab#667 gate and reached
main through Open-Finance-Lab#662's follow-up; since Open-Finance-Lab#667 the gate sizes it as a share
of the position (0.25 sells a quarter, 1 or more sells all). Comment
only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(backtests): an LLM action with trailing whitespace ("buy ") is silently dropped

2 participants