Skip to content

fix(backtests): an LLM sell sells the shares it asked for (#661) - #667

Merged
FlyM1ss merged 4 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-sell-size
Oct 30, 2026
Merged

FlyM1ss merged 4 commits into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-sell-size

Conversation

@subidhkhanal

@subidhkhanal subidhkhanal commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #661.

The bug

In an LLM backtest every SELL closed the whole position. The prompt asks the model for the shares to sell, but the sell branch of make_trading_decision_with_llm (domain/backtesting/portfolio_manager.py) never read position_size. "Sell 10 AAPL" while holding 500 sold 500, so every trim became a full exit.

The fix

Sell branch (portfolio_manager.py). A sell is sized like this:

  • position_size > 0 → that many shares, capped at the holding and at the T+1-sellable quantity. When capped, the action carries requested_shares and the T+1 deferral is recorded against the capped request.
  • position_size missing or 0 → the whole position, as before. This is the same convention as actions_to_executable.
  • unparseable, negative, or under one share (US int(0.5) = 0) → that action is skipped with a log line. It is never read as "sell everything".

The buy branch's position_size coercion moved into _coerce_position_size, which both branches use. Buy behaviour is unchanged.

Pipeline risk actions (infrastructure/llm/pipeline_runner.py). risk_actions turned size_pct into position_size = round(size_pct*100), a percentage the old gate ignored. Read as shares, a full stop-loss on 500 would have sold 100. Risk sells now send position_size: 0 plus size_fraction: size_pct. The gate turns that into floor(fraction × held), rounded down to whole lots on A-shares; a fraction ≥ 1 is a full exit.

Decision tape (domain/backtesting/decision_tape.py).

  • size_fraction is added to the intent allow-list, so a replay hands the gate the same order.
  • Sell sizes are now compared like buy sizes. A sell capped by the holding or T+1 counts as a gate rewrite; a size-0 or size_fraction sell does not.

Tests

  • New tests/backtesting/test_llm_sell_sizing.py (21 cases): sell 10 of 500 → 10; missing/0/None → 500; 900 → 500 with requested_shares; unparseable, negative, and under-one-share sizes are skipped; size_fraction 0.25/1.0/1.5; A-share lot rounding; T+1 cap and deferral.
  • Updated to the new behaviour:
    • test_pipeline_runner.py: the risk action carries size_fraction.
    • test_decision_tape.py: the risk intent shape and the rewrite rules for sells.
    • test_decision_tape_engine.py: a trim of 1 out of 4 sells 1.
  • Conformance baselines re-recorded for C01/DL and C04/DL only. In C01, "sell 5" now fills 5, matching the reference. In C04, "sell -5" no longer liquidates.
  • Full backend suite: same 130 failures + 1 error as clean main on the same machine, compared name by name (local environment failures). No new failures; 19 more passing.

Behaviour change to be aware of

  • LLM sells now trim instead of always exiting, so leaderboard and backtest curves made after merge are not comparable to older runs wherever the model trimmed.
  • Decision tapes recorded before this change don't replay sells exactly. Old risk sells stored size_pct*100 as a share count.

Known limits (left for follow-ups)

Maintainer follow-up (16aa538, 982f94a, b5992c0)

Merged main (including #673, and #662 in b5992c0), then fixed what the review found:

  • One sizing rule. validator.requested_sell_shares sizes a SELL for the backtest gate and for actions_to_executable (external agents). Only a missing, null or 0 size means the whole position. A negative, fractional (US), non-numeric, bool or non-finite position_size / size_fraction is refused (-0.5 and false used to liquidate), and so is a trim that rounds to no shares. 0.57 × 100 sells 57, not 56, and a 400-digit size_fraction no longer raises.
  • Refusals are recorded. A refused SELL lands on order_events / rejected_orders as invalid_quantity, collapsed like an executor rejection. Conformance C04/DL's o2 (sell -5) now matches the reference.
  • Prompts. SAFE_TRADING_PROMPT and the custom-strategy contract say a SELL's position_size is the shares to sell (0 = all), and the templates no longer show 1. The starter instruction (defaults.py, app.js, the 7 marketplace cards) says to set qty to all the shares held; the old text still counts as the default. The pipeline's EXECUTION RULES state the rule on every run, for agents created with the old sentence. app.js?v=163.
  • Provenance. Runs record llm_sell_sizing. Leaderboard repeats sized by two rules never pool (_SAMPLE_CONFIG_KEYS). A Live checkpoint without the key is read as the old rule, so its month is replayed, not spliced.
  • Decision tape v2. gate_rewrote pairs sizes per symbol and side as a multiset, so a split sell is not called a rewrite.
  • SECURITY.md, the conformance inventory and the tape and conformance design docs are updated.

Of the known limits above, the prompt wording, non-numeric size_pct (via #673), order events for skipped sells, and the duplicated sell rule (the sell half of #665) are addressed. size_pct ≥ 1 still means a full exit, and no prompt defines its unit yet.

Live board on deploy: the first advance after this ships finds the month's checkpoints without llm_sell_sizing and replays the month to date for each LLM entry (WARNING: live.config_changed keys=llm_sell_sizing). That is a one-off set of billed calls, and the month is redrawn under the new rule. Deployed after the Oct 30 advance, October stays as recorded and November opens under the new rule.

Full backend suite on Linux: 7240 passed. The 5 test_robinhood_mcp_transport failures are the local mcp 1.28.1 against the 1.30.0 pin; this PR does not touch that path.

🤖 Generated with Claude Code

…ce-Lab#661)

The sell branch of the LLM decision loop ignored position_size and always
closed the whole position, so every trim the model asked for became a full
exit. It now sells position_size shares, capped at the holding and at the
T+1-sellable quantity (recording requested_shares and the deferral when it
is capped). A missing or zero size still sells the whole position. An
unparseable, negative, or under-one-share size skips that action only.

Pipeline risk actions sent size_pct as position_size = size_pct*100, a
percentage the old gate ignored; read as shares it would have sold 50 of
500 for "sell 50%". They now send size_fraction, which the gate turns into
floor(fraction x held), rounded down to whole lots on A-shares.

The decision tape records size_fraction so replays match, and compares sell
sizes like buy sizes: a sell capped by the holding or T+1 is a rewrite.
Conformance C01/DL and C04/DL baselines re-recorded (sell 5 now fills 5).
@vercel

vercel Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel.

A member of the Team first needs to authorize it.

FlyM1ss and others added 3 commits October 30, 2026 01:09
One conflict, in pipeline_output_to_decision's risk branch. Kept main's
reading of size_pct (Open-Finance-Lab#673: a missing size sells all, an unusable or
non-positive one holds) and confidence; a SELL now carries size_pct as
size_fraction with position_size 0, as this branch intends, instead of
main's round(size_pct * 100) share count. The risk test main added is
updated to the fraction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rything

Review follow-ups to the sell-sizing change (Open-Finance-Lab#661), on top of main (Open-Finance-Lab#673):

- One sizing rule, validator.requested_sell_shares, for the gate and for
  actions_to_executable. Only a missing, null or 0 size means the whole
  position. A negative, fractional, non-numeric, bool or non-finite
  position_size or size_fraction is refused rather than read as "sell all"
  (-0.5 and false used to liquidate). A trim that rounds to no shares is
  refused too. The sign is read before anything truncates, a 400-digit
  size_fraction no longer raises OverflowError, and 0.57 x 100 sells 57,
  not 56.
- A refused SELL is recorded as an invalid_quantity rejection on
  order_events and rejected_orders, collapsed like an executor rejection.
  It used to be only a print in a subprocess. Conformance C04/DL's o2
  (sell -5) now matches the reference, and the baseline is re-recorded.
- Prompts: SAFE_TRADING_PROMPT and the custom-strategy contract said
  "otherwise use 1" and showed position_size 1. They now say a SELL's
  position_size is the shares to sell (0 = all). The starter instruction
  (defaults.py, app.js, 7 marketplace cards) says to set qty to all the
  shares held. The pipeline's EXECUTION RULES state the rule on every run,
  for agents seeded with the old sentence. The old wording still counts as
  the default (PRIOR_STARTER_INSTRUCTIONS). app.js v=163.
- Runs record llm_sell_sizing. A leaderboard pool never mixes rows of two
  rules (_SAMPLE_CONFIG_KEYS). A Live checkpoint without the key is read
  as the old rule, so its month is replayed, not spliced.
- Decision tape v2. gate_rewrote pairs sizes as a multiset per symbol and
  side, so a split sell is no longer called a rewrite.
- SECURITY.md table, conformance inventory and the tape and conformance
  design docs updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts with Open-Finance-Lab#662 (normalize_action), all combined:
- SECURITY.md: main's count-cap Action count row, this branch's split
  BUY/SELL per-order size rows.
- decision_tape._order_triples: main's normalize_symbol, this branch's
  sell sizes and unsorted triples (gate_rewrote pairs them as a multiset).
- portfolio_manager imports: both model_number and normalize_action.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@FlyM1ss

FlyM1ss commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Post-merge follow-ups:

FlyM1ss added a commit to subidhkhanal/AgenticTrading that referenced this pull request Oct 30, 2026
…confidence-range

Conflicts resolved:
- portfolio_manager.py: kept main's `_coerce_position_size` and dropped this
  branch's identical `_requested_buy_shares`; kept both `recognised_any` (Open-Finance-Lab#658)
  and `skipped_for_confidence`. Open-Finance-Lab#667's sell branch formatted
  `{confidence:.0%}`, which raises on the unreadable confidence this gate lets
  a sell trade on; it now uses `conf_label`.
- engine.py, leaderboard/service.py and their metadata tests: record both
  `llm_gate_version` and `llm_sell_sizing`.
- validator.py: Open-Finance-Lab#667's sell-sizing block, then the public
  `confidence_in_range`.
- backtest_custom_algo.py, ai_hedge_fund/adapter.py: Open-Finance-Lab#662's
  normalize_symbol/normalize_action alongside read_confidence and
  confidence_in_range.
- SECURITY.md: the versioned-gate paragraph now names `llm_sell_sizing` and
  the reply readings; the Trading Algo row lists all three shared readers.

LLM_GATE_VERSION's history notes that v2, the first recorded version, also
covers Open-Finance-Lab#658 and Open-Finance-Lab#661.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
computedocean pushed a commit to computedocean/AgenticTrading that referenced this pull request Oct 31, 2026
The risk-branch comment in pipeline_output_to_decision said any
positive size_pct sells the whole position because the gate does not
size a sell. That was written against the pre-Open-Finance-Lab#667 gate and reached
main through Open-Finance-Lab#662's follow-up; since Open-Finance-Lab#667 the gate sizes it as a share
of the position (0.25 sells a quarter, 1 or more sells all). Comment
only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(backtests): an LLM "sell 10" sells the whole position

2 participants