Repository navigation
fix(backtests): never auto-size an LLM buy whose requested size is under one share or not positive - #675
Open
subidhkhanal wants to merge 1 commit into
Conversation
…der one share or not positive The buy branch read 'no size' off the coerced value, so int() turning 0.6 or -0.5 into 0 made the gate size the buy from confidence: a model asking for 0.6 of a share got 18 shares in its name. Whether a size was given is now read off the raw value: only a missing, null, empty or zero size is confidence-sized; a given size under one share or not positive is skipped with its reason, and a boolean is refused as unparseable. SECURITY.md's per-order size row and the S01/DL conformance note are updated.
Contributor
|
@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
In
make_trading_decision_with_llm(domain/backtesting/portfolio_manager.py), the buy branch coerces the model'sposition_sizewithint()on US runs and then runsif shares == 0:to size the buy from confidence (2% of equity × confidence).shares == 0meant two different things:int(0.6),int(-0.5),int(False)).So a model asking for 0.6 of a share got a buy the engine chose, and the step still counted as model-driven. Reproduced on
mainwith AAPL at $100 and confidence 0.9:main0.6,0.99-0.5falsetrue-3On a $500 stock,
0.6bought 3 shares ($1,500).The fix
Whether the model gave a size is now read off the raw value, before coercion:
null,""or0): sized from confidence, unchanged.position_size 0.6 rounds to 0 shares, orinvalid position_size -3, not positive.Unchanged:
1→ 1 share and10.7→ 10;0.6reaches the executor's lot check, as before.Docs:
dashboard/backend/SECURITY.md: the per-order size row (the one copy of the LLM loop's limits) describes the new rule.tests/conformance/cases.py: the S01/DL note is updated. The conformance baseline itself did not change.Tests
New cases in
tests/backtesting/test_portfolio_manager_move.py:0.6,0.99→ skipped, "rounds to 0 shares";-0.5,-3→ skipped, "not positive";true,false→ skipped, unparseable;0,null,""→ still confidence-sized;10.7→ 10.The skip cases failed on
mainas in the table above and pass with the fix. Focused suites (tests/backtesting,tests/domain/backtesting,tests/infrastructure/llm,tests/conformance): 1142 passed, 0 failed.Out of scope
"10.0"are still refused as unparseable: fix(backtests): a pipeline reply with a non-numeric confidence or qty aborts a paid run and loses the step's usage #663.dashboard/scripts/backtest_custom_algo.py) has the sameint()truncation into confidence sizing. CLAUDE.md documents that loop as not hardened onto the shared helpers, so it is left for a separate change.Note
#667 and #674 also edit the buy/sell gate in
portfolio_manager.py. Whichever merges later may need a small conflict resolved; the changes do not overlap in logic.