Skip to content

fix(sleep): keep the gate non-empty when every task hashes into test - #276

Closed
hylin (linhongyu510) wants to merge 1 commit into
microsoft:mainfrom
linhongyu510:fix/assign-splits-all-test-degenerate
Closed

fix(sleep): keep the gate non-empty when every task hashes into test#276
hylin (linhongyu510) wants to merge 1 commit into
microsoft:mainfrom
linhongyu510:fix/assign-splits-all-test-degenerate

Conversation

@linhongyu510

Copy link
Copy Markdown

Fixes #271.

The gap

assign_splits in skillopt_sleep/mine.py says right in its own comment
"Guarantee val (the gate) is non-empty when we have >=2 real tasks", but
_promote_one only pulls from train (to top up val) or from val (to top
up train), never from test. If every real task's hash bucket lands in
[val_cut, test_cut), both train and val start empty, so both guarantee
calls have nothing to promote and silently no-op.

run_sleep_cycle then finishes with gate_action='reject', edits=0, no
error, no warning, and holdout_leaked does not flag it either. The night
looks like it ran and learned nothing, with no signal saying why.

As #271 notes, this is not contrived: for a small nightly batch (2-5 tasks is
realistic for a solo user), any test_fraction above roughly 0.5 makes it a
matter of when, not if.

Reproduction on 79124b37

from collections import Counter
from skillopt_sleep.mine import assign_splits
from skillopt_sleep.types import TaskRecord

tasks = [
    TaskRecord(id=f"task-{i}", project="/p", intent="x",
               reference_kind="exact", reference=f"a{i}", split="", origin="real")
    for i in range(5)
]
assign_splits(tasks, val_fraction=0.10, test_fraction=0.80, seed=42)
print(Counter(t.split for t in tasks))

Before: Counter({'test': 5}) — val and train both empty, no warning.
After: Counter({'test': 3, 'val': 1, 'train': 1}) plus a WARNING naming the
fractions and seed.

The change

Two small, additive adjustments in assign_splits:

  1. _promote_one now returns whether it promoted anything, so a no-op is
    distinguishable from a promotion.
  2. Each guarantee keeps its existing preferred source and only reaches into
    test when that source is empty. #235's stability guarantee is untouched
    in the normal case — hash-assigned test tasks are reassigned only in the
    degenerate case where the alternative is a dead cycle.

Topping up train prefers test over a single-task val, because taking
the only val row would re-empty the gate that was just filled. When there is no
test slice at all, it still falls back to val exactly as before.

Borrowing does spend a held-out task, so it is logged via
logging.getLogger("skillopt_sleep").warning(...) — the module's existing
convention — naming the task count, both fractions and the seed, and pointing
at test_fraction as the knob to lower.

This is the shape #271 proposed. I also hit one case the issue does not
mention: when dream tasks are present they occupy train, so the second
guarantee is already satisfied and only the val guarantee fires. Filling val
from test still has to happen there, and val stays real-only.

Tests

Added to Pass1ApproachBAssignSplitsInvariants, next to the existing #235
stability test:

  • test_val_guaranteed_when_every_real_task_hashes_into_test — subtests over
    2..10 real tasks
  • test_degenerate_split_warns_that_test_was_spent
  • test_dream_train_does_not_mask_an_empty_gate — val stays real-only; dream
    stays train
  • test_normal_split_does_not_borrow_from_test — asserts no logger is even
    constructed on the healthy path
  • test_single_real_task_is_left_alone — the >=2 guard still applies

Reverting mine.py alone turns 7 of these red, so they pin the behavior rather
than restate it.

Verification

  • python -m pytest tests/1501 passed, 12 skipped, 362 subtests passed
  • python -m ruff check skillopt_sleep/mine.py — clean

tests/test_split_hardening_2x3.py has one pre-existing ruff I001 import-order
finding on main; I left it alone to keep this diff minimal.

assign_splits promises "Guarantee val (the gate) is non-empty when we have
>=2 real tasks", but _promote_one only pulls from train (to top up val) or
from val (to top up train), never from test. When every real task's hash
bucket lands in [val_cut, test_cut) both train and val start empty, so both
guarantee calls have nothing to promote and silently no-op.

run_sleep_cycle then finishes with gate_action='reject' and edits=0: no
error, no warning, and holdout_leaked does not flag it either. For a small
nightly batch any test_fraction above roughly 0.5 makes this a matter of
when, not if.

Fall back to test for the promotion only when the preferred source is empty,
so the stability guarantee microsoft#235 settled is untouched in the normal case and
hash-assigned test tasks are only reassigned in the degenerate one. Topping
up train now prefers test over a single-task val, so filling train cannot
re-empty the gate it just filled. Borrowing spends a held-out task, so it is
logged the way consolidate.py already logs holdout_leaked.

Tests cover the degenerate split for 2..10 real tasks, the warning, dream
tasks in train not masking an empty gate, the normal path not borrowing, and
the single-real-task guard.
@linhongyu510

Copy link
Copy Markdown
Author

Closing as a duplicate — I missed #272, which was opened three days earlier for the same issue, touches the same two files, and lands the same fallback (prefer the non-test pool, reach into test only when both required pools cannot otherwise be filled, and warn when test coverage is spent). Apologies for the noise; #272 should be the one reviewed.

One detail from my run that may be worth folding into #272 if it is not already covered: when dream tasks are present they occupy train, so the train guarantee is satisfied and only the val guarantee fires. #272 gates the val top-up on train_count >= 2 counting all tasks including dream, so a batch with dream rows plus an all-test real hash should be worth an explicit assertion that val is still filled from a real task and never from a dream row.

@linhongyu510

Copy link
Copy Markdown
Author

Duplicate of #272.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

assign_splits's own non-empty-val guarantee doesn't fire when every real task hashes into test

1 participant