Repository navigation
fix(backtests): an LLM sell sells the shares it asked for (#661) - #667
Merged
Merged
Conversation
…ce-Lab#661) The sell branch of the LLM decision loop ignored position_size and always closed the whole position, so every trim the model asked for became a full exit. It now sells position_size shares, capped at the holding and at the T+1-sellable quantity (recording requested_shares and the deferral when it is capped). A missing or zero size still sells the whole position. An unparseable, negative, or under-one-share size skips that action only. Pipeline risk actions sent size_pct as position_size = size_pct*100, a percentage the old gate ignored; read as shares it would have sold 50 of 500 for "sell 50%". They now send size_fraction, which the gate turns into floor(fraction x held), rounded down to whole lots on A-shares. The decision tape records size_fraction so replays match, and compares sell sizes like buy sizes: a sell capped by the holding or T+1 is a rewrite. Conformance C01/DL and C04/DL baselines re-recorded (sell 5 now fills 5).
Contributor
|
@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel. A member of the Team first needs to authorize it. |
One conflict, in pipeline_output_to_decision's risk branch. Kept main's reading of size_pct (Open-Finance-Lab#673: a missing size sells all, an unusable or non-positive one holds) and confidence; a SELL now carries size_pct as size_fraction with position_size 0, as this branch intends, instead of main's round(size_pct * 100) share count. The risk test main added is updated to the fraction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rything Review follow-ups to the sell-sizing change (Open-Finance-Lab#661), on top of main (Open-Finance-Lab#673): - One sizing rule, validator.requested_sell_shares, for the gate and for actions_to_executable. Only a missing, null or 0 size means the whole position. A negative, fractional, non-numeric, bool or non-finite position_size or size_fraction is refused rather than read as "sell all" (-0.5 and false used to liquidate). A trim that rounds to no shares is refused too. The sign is read before anything truncates, a 400-digit size_fraction no longer raises OverflowError, and 0.57 x 100 sells 57, not 56. - A refused SELL is recorded as an invalid_quantity rejection on order_events and rejected_orders, collapsed like an executor rejection. It used to be only a print in a subprocess. Conformance C04/DL's o2 (sell -5) now matches the reference, and the baseline is re-recorded. - Prompts: SAFE_TRADING_PROMPT and the custom-strategy contract said "otherwise use 1" and showed position_size 1. They now say a SELL's position_size is the shares to sell (0 = all). The starter instruction (defaults.py, app.js, 7 marketplace cards) says to set qty to all the shares held. The pipeline's EXECUTION RULES state the rule on every run, for agents seeded with the old sentence. The old wording still counts as the default (PRIOR_STARTER_INSTRUCTIONS). app.js v=163. - Runs record llm_sell_sizing. A leaderboard pool never mixes rows of two rules (_SAMPLE_CONFIG_KEYS). A Live checkpoint without the key is read as the old rule, so its month is replayed, not spliced. - Decision tape v2. gate_rewrote pairs sizes as a multiset per symbol and side, so a split sell is no longer called a rewrite. - SECURITY.md table, conformance inventory and the tape and conformance design docs updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts with Open-Finance-Lab#662 (normalize_action), all combined: - SECURITY.md: main's count-cap Action count row, this branch's split BUY/SELL per-order size rows. - decision_tape._order_triples: main's normalize_symbol, this branch's sell sizes and unsorted triples (gate_rewrote pairs them as a multiset). - portfolio_manager imports: both model_number and normalize_action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 30, 2026
Collaborator
FlyM1ss
added a commit
to subidhkhanal/AgenticTrading
that referenced
this pull request
Oct 30, 2026
…confidence-range Conflicts resolved: - portfolio_manager.py: kept main's `_coerce_position_size` and dropped this branch's identical `_requested_buy_shares`; kept both `recognised_any` (Open-Finance-Lab#658) and `skipped_for_confidence`. Open-Finance-Lab#667's sell branch formatted `{confidence:.0%}`, which raises on the unreadable confidence this gate lets a sell trade on; it now uses `conf_label`. - engine.py, leaderboard/service.py and their metadata tests: record both `llm_gate_version` and `llm_sell_sizing`. - validator.py: Open-Finance-Lab#667's sell-sizing block, then the public `confidence_in_range`. - backtest_custom_algo.py, ai_hedge_fund/adapter.py: Open-Finance-Lab#662's normalize_symbol/normalize_action alongside read_confidence and confidence_in_range. - SECURITY.md: the versioned-gate paragraph now names `llm_sell_sizing` and the reply readings; the Trading Algo row lists all three shared readers. LLM_GATE_VERSION's history notes that v2, the first recorded version, also covers Open-Finance-Lab#658 and Open-Finance-Lab#661. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
computedocean
pushed a commit
to computedocean/AgenticTrading
that referenced
this pull request
Oct 31, 2026
The risk-branch comment in pipeline_output_to_decision said any positive size_pct sells the whole position because the gate does not size a sell. That was written against the pre-Open-Finance-Lab#667 gate and reached main through Open-Finance-Lab#662's follow-up; since Open-Finance-Lab#667 the gate sizes it as a share of the position (0.25 sells a quarter, 1 or more sells all). Comment only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #661.
The bug
In an LLM backtest every SELL closed the whole position. The prompt asks the model for the shares to sell, but the sell branch of
make_trading_decision_with_llm(domain/backtesting/portfolio_manager.py) never readposition_size. "Sell 10 AAPL" while holding 500 sold 500, so every trim became a full exit.The fix
Sell branch (
portfolio_manager.py). A sell is sized like this:position_size > 0→ that many shares, capped at the holding and at the T+1-sellable quantity. When capped, the action carriesrequested_sharesand the T+1 deferral is recorded against the capped request.position_sizemissing or 0 → the whole position, as before. This is the same convention asactions_to_executable.int(0.5)= 0) → that action is skipped with a log line. It is never read as "sell everything".The buy branch's
position_sizecoercion moved into_coerce_position_size, which both branches use. Buy behaviour is unchanged.Pipeline risk actions (
infrastructure/llm/pipeline_runner.py).risk_actionsturnedsize_pctintoposition_size = round(size_pct*100), a percentage the old gate ignored. Read as shares, a full stop-loss on 500 would have sold 100. Risk sells now sendposition_size: 0plussize_fraction: size_pct. The gate turns that intofloor(fraction × held), rounded down to whole lots on A-shares; a fraction ≥ 1 is a full exit.Decision tape (
domain/backtesting/decision_tape.py).size_fractionis added to the intent allow-list, so a replay hands the gate the same order.size_fractionsell does not.Tests
tests/backtesting/test_llm_sell_sizing.py(21 cases): sell 10 of 500 → 10; missing/0/None → 500; 900 → 500 withrequested_shares; unparseable, negative, and under-one-share sizes are skipped;size_fraction0.25/1.0/1.5; A-share lot rounding; T+1 cap and deferral.test_pipeline_runner.py: the risk action carriessize_fraction.test_decision_tape.py: the risk intent shape and the rewrite rules for sells.test_decision_tape_engine.py: a trim of 1 out of 4 sells 1.mainon the same machine, compared name by name (local environment failures). No new failures; 19 more passing.Behaviour change to be aware of
size_pct*100as a share count.Known limits (left for follow-ups)
"position_size": 1. A model that falls back to 1 now sells one share instead of closing the position. The prompt is deliberately unchanged here.size_pctunits. A value of 0, ≥ 1 (e.g.25meaning 25%) or negative still means a full exit, as before. No prompt defines the unit.size_pct.float(size_pct)still raises on non-numeric input (fix(backtests): a pipeline reply with a non-numeric confidence or qty aborts a paid run and loses the step's usage #663).gate_rewrote.validator.actions_to_executable. Unifying the two is refactor(llm): check the AI's trading answer with one shared schema — today two code paths read it differently #665.Maintainer follow-up (16aa538, 982f94a, b5992c0)
Merged
main(including #673, and #662 in b5992c0), then fixed what the review found:validator.requested_sell_sharessizes a SELL for the backtest gate and foractions_to_executable(external agents). Only a missing, null or 0 size means the whole position. A negative, fractional (US), non-numeric, bool or non-finiteposition_size/size_fractionis refused (-0.5andfalseused to liquidate), and so is a trim that rounds to no shares.0.57 × 100sells 57, not 56, and a 400-digitsize_fractionno longer raises.order_events/rejected_ordersasinvalid_quantity, collapsed like an executor rejection. Conformance C04/DL's o2 (sell -5) now matches the reference.position_sizeis the shares to sell (0 = all), and the templates no longer show1. The starter instruction (defaults.py,app.js, the 7 marketplace cards) says to set qty to all the shares held; the old text still counts as the default. The pipeline's EXECUTION RULES state the rule on every run, for agents created with the old sentence.app.js?v=163.llm_sell_sizing. Leaderboard repeats sized by two rules never pool (_SAMPLE_CONFIG_KEYS). A Live checkpoint without the key is read as the old rule, so its month is replayed, not spliced.gate_rewrotepairs sizes per symbol and side as a multiset, so a split sell is not called a rewrite.SECURITY.md, the conformance inventory and the tape and conformance design docs are updated.Of the known limits above, the prompt wording, non-numeric
size_pct(via #673), order events for skipped sells, and the duplicated sell rule (the sell half of #665) are addressed.size_pct≥ 1 still means a full exit, and no prompt defines its unit yet.Live board on deploy: the first advance after this ships finds the month's checkpoints without
llm_sell_sizingand replays the month to date for each LLM entry (WARNING: live.config_changed keys=llm_sell_sizing). That is a one-off set of billed calls, and the month is redrawn under the new rule. Deployed after the Oct 30 advance, October stays as recorded and November opens under the new rule.Full backend suite on Linux: 7240 passed. The 5
test_robinhood_mcp_transportfailures are the localmcp1.28.1 against the 1.30.0 pin; this PR does not touch that path.🤖 Generated with Claude Code