Skip to content

fix(backtests): never auto-size an LLM buy whose requested size is under one share or not positive - #675

Open
subidhkhanal wants to merge 1 commit into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-fractional-buy-size
Open

subidhkhanal wants to merge 1 commit into
Open-Finance-Lab:mainfrom
subidhkhanal:fix/llm-fractional-buy-size

Conversation

@subidhkhanal

Copy link
Copy Markdown
Contributor

The bug

In make_trading_decision_with_llm (domain/backtesting/portfolio_manager.py), the buy branch coerces the model's position_size with int() on US runs and then runs if shares == 0: to size the buy from confidence (2% of equity × confidence). shares == 0 meant two different things:

  • the model gave no size (intended: auto-size);
  • the model's size truncated to 0 (int(0.6), int(-0.5), int(False)).

So a model asking for 0.6 of a share got a buy the engine chose, and the step still counted as model-driven. Reproduced on main with AAPL at $100 and confidence 0.9:

AI sends Bought on main
0.6, 0.99 18 shares
-0.5 18 shares
false 18 shares
true 1 share
-3 skipped, but logged as "insufficient cash: need $-300"

On a $500 stock, 0.6 bought 3 shares ($1,500).

The fix

Whether the model gave a size is now read off the raw value, before coercion:

  • No size (missing, null, "" or 0): sized from confidence, unchanged.
  • A size was given: it must coerce to at least one share. Otherwise that buy is skipped with its reason:
    • position_size 0.6 rounds to 0 shares, or
    • invalid position_size -3, not positive.
  • A boolean is refused as unparseable.

Unchanged:

  • 1 → 1 share and 10.7 → 10;
  • unparseable and non-finite sizes are still skipped;
  • A-share runs keep the size as a float, so 0.6 reaches the executor's lot check, as before.

Docs:

  • dashboard/backend/SECURITY.md: the per-order size row (the one copy of the LLM loop's limits) describes the new rule.
  • tests/conformance/cases.py: the S01/DL note is updated. The conformance baseline itself did not change.

Tests

New cases in tests/backtesting/test_portfolio_manager_move.py:

  • 0.6, 0.99 → skipped, "rounds to 0 shares";
  • -0.5, -3 → skipped, "not positive";
  • true, false → skipped, unparseable;
  • missing, 0, null, "" → still confidence-sized;
  • 10.7 → 10.

The skip cases failed on main as in the table above and pass with the fix. Focused suites (tests/backtesting, tests/domain/backtesting, tests/infrastructure/llm, tests/conformance): 1142 passed, 0 failed.

Out of scope

Note

#667 and #674 also edit the buy/sell gate in portfolio_manager.py. Whichever merges later may need a small conflict resolved; the changes do not overlap in logic.

…der one share or not positive

The buy branch read 'no size' off the coerced value, so int() turning 0.6 or -0.5 into 0 made the gate size the buy from confidence: a model asking for 0.6 of a share got 18 shares in its name. Whether a size was given is now read off the raw value: only a missing, null, empty or zero size is confidence-sized; a given size under one share or not positive is skipped with its reason, and a boolean is refused as unparseable. SECURITY.md's per-order size row and the S01/DL conformance note are updated.
@vercel

vercel Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

@subidhkhanal is attempting to deploy a commit to the allan-feng's projects Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant