feat(evals): optional QualityWindow frame selection for VQA generate by moshiii-wei · Pull Request #4566 · dimensionalOS/dimos

moshiii-wei · 2026-10-14T14:48:15Z

What is this feature?

Adds optional quality-based frame picking for dimos evals vqa generate in range mode. Instead of a custom sharp-stride neighborhood, generation reuses mem QualityWindow on Image.sharpness (same path as loop-closure / semantic-peak picking), with an optional brightness floor.

Uniform --stride remains available; --quality-window replaces stride when you want frequency-preserving sharpest-frame selection.

Why do we need this?

Blind stride can land on blurry frames. The platform already measures sharpness/brightness and keeps update frequency via QualityWindow / sharpness_barrier — VQA generate should use that instead of a parallel selection algorithm.

Comparison experiment (go2_short)

A/B on go2_short color_image [0, 855) (~60s @ ~14 Hz). QualityWindow used window=1.0s, min_brightness=0.1; uniform stride was matched to the same count (stride=15, n=57).

Frame selection

metric	stride	QualityWindow
mean sharpness	0.796	0.834
median sharpness	0.808	0.839
min sharpness	0.478	0.646
frac sharpness < 0.5	1.8%	0%
QW sharper vs nearest-in-time stride frame	—	84.2% of pairs
mean sharpness Δ (QW − stride)	—	+0.022
exact index overlap	—	0 / 57

Eval-case yield (image-only families)

Ran the real _generate_frame path (OpenAI author → Moondream GT) on 12 evenly spaced frames per arm. Families: presence, horizontal_direction, object_count (no masks / pointcloud — go2_short has no camera_info/tf).

metric	stride	QualityWindow
answered cases	23	33 (+43%)
rejected proposals	10	4
reject rate	30.3%	10.8%
answered cases / frame	1.92	2.75
proposals / frame	2.75	3.08

By family (answered / rejected):

family	stride	QualityWindow
presence	11 / 2	14 / 0
object_count	10 / 2	13 / 0
horizontal_direction	2 / 6	6 / 4

Takeaway: sharper picks cut family rejects (~3× lower reject rate) and increased kept eval cases on this sample. Mask / LiDAR families not measured here.

How to Test

# unit tests (no LFS / no models beyond what these suites mock)
uv run --python 3.11 pytest \
  dimos/evals/vqa/test_contracts.py \
  dimos/evals/vqa/test_pointcloud_frame.py \
  dimos/cli/test_vqa.py -q

# help surfaces the new flags
uv run --python 3.11 dimos evals vqa generate --help | rg 'quality-window|min-brightness'

# optional: quality-window generate (needs calibrated recording with camera_info + tf)
uv run --python 3.11 dimos evals vqa generate /path/to/calibrated-recording.db \
  --start 0 --stop 100 --quality-window 0.5 --min-brightness 0.1

Which issue(s) does this PR close?

NA

Checklist

I have read and approved the CLA.

Keep uniform start/stop/stride anchors, then pick the sharpest nearby color_image in a stride-aware neighborhood so motion-blurred samples are less likely to enter deterministic VQA cases.

github-actions

go2 replay realtime (arm64)

Details

Benchmark suite	Current: `ce0d609`	Previous: `3faae1f`	Ratio
`peak memory`	`1585.992` MB	`1589.676` MB	`1.00`
`peak threads`	`356` threads	`360` threads	`0.99`
`network (transport)`	`2722.098` MB	`2722.636` MB	`1.00`
`disk write`	`1.461` MB	`1.734` MB	`0.84`
`instructions`	`115.648` G	`115.588` G	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

codecov · 2026-10-14T14:56:28Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
6227	1	6226	86

View the top 1 failed test(s) by shortest run time

dimos.codebase_checks.test_no_dunder_new::test_no_dunder_new

Stack Traces | 2.51s run time

def test_no_dunder_new() -> None:
        """Fail if any test file calls `__new__` to bypass `__init__`."""
        dimos_dir = DIMOS_PROJECT_ROOT / "dimos"
        hits = find_dunder_new_calls()
        if hits:
            listing = "\n".join(
                f"  - {p.relative_to(dimos_dir)}:{lineno}: {line.strip()}" for p, lineno, line in hits
            )
>           raise AssertionError(
                f"Found __new__ call(s) in test files:\n{listing}\n\n"
                "Tests must construct objects with the real constructor: __init__ is "
                "code under test too, and an object assembled by hand silently rots "
                "when the constructor changes. If __init__ does heavy work, mock the "
                "collaborators it needs instead of skipping it. Only if that is truly "
                "impossible, add the call to the WHITELIST in "
                "dimos/codebase_checks/test_no_dunder_new.py."
            )
E           AssertionError: Found __new__ call(s) in test files:
E             - evals/vqa/test_pointcloud_frame.py:343: loader = PointCloudFrameLoader.__new__(PointCloudFrameLoader)
E           
E           Tests must construct objects with the real constructor: __init__ is code under test too, and an object assembled by hand silently rots when the constructor changes. If __init__ does heavy work, mock the collaborators it needs instead of skipping it. Only if that is truly impossible, add the call to the WHITELIST in dimos/codebase_checks/test_no_dunder_new.py.

dimos_dir  = PosixPath('.../dimos/dimos/dimos')
hits       = [(PosixPath('.../dimos/dimos/dimos/evals/vqa/test_pointcloud_frame.py'), 343, '        loader = PointCloudFrameLoader.__new__(PointCloudFrameLoader)')]
listing    = '  - evals/vqa/test_pointcloud_frame.py:343: loader = PointCloudFrameLoader.__new__(PointCloudFrameLoader)'

dimos/codebase_checks/test_no_dunder_new.py:67: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

greptile-apps · 2026-10-14T14:56:42Z

Confidence Score: 4/5

[Medium impact] Adds optional frame selection refinement to VQA dataset generation.

Do not merge until sharp-stride respects the requested start and exclusive stop bounds. The separate regression-coverage concern is non-blocking.

Findings

Frames escape the requested range ▶
Generation path remains untested ▶

T-Rex evidence

Scoped generation reproduction script

Executes real generation selection and consumes its frame iterator with fake supporting services, asserting the exact selected indices at both request boundaries.

Commands and output capture script

Runs the reproduction with sharp-stride disabled and enabled and records each command, working directory, exit code, and observed output for comparison.

Generation output with sharp-stride disabled

Captured the baseline generation runs for both score profiles; only frame 100 was loaded and forwarded, respecting the requested range.

Generation output with sharp-stride enabled

Captured generation selecting and forwarding frames 104 and 96 in the respective score profiles, confirming that both request boundaries can be violated.

Executable baseline and mutation harness

This executed Python harness applies the in-memory mutation, runs the existing tests, and probes generation with deterministic doubles, making the coverage comparison reproducible.

Executed commands and evidence capture script

This executed driver records command, working directory, exit code, and observed output for inspection and the single before/after pair.

Scoped selector, generation, and CLI test inspection

Numbered source excerpts and search output show direct selector tests and CLI tests without sharp-option checks, locating the coverage gap.

Tests and generation probe before mutation

The unmodified run records 83 passes and two failures while the separate generation probe emits selected indices (12, 22), establishing the baseline.

Tests and generation probe after disconnecting selected indices

The mutated run records the same test outcomes while generation emits anchors (10, 20), showing that the existing tests do not detect the disconnection.

Tracked source status after validation

The final git diff check exits successfully with no diff, confirming that validation left tracked sources unchanged.

Evidence from the check

Executes real generation selection and consumes its frame iterator with fake supporting services, asserting the exact selected indices at both request boundaries.

Evidence from the check

Runs the reproduction with sharp-stride disabled and enabled and records each command, working directory, exit code, and observed output for comparison.

Command output from the check

Captured the baseline generation runs for both score profiles; only frame 100 was loaded and forwarded, respecting the requested range.

Command output from the check

Captured generation selecting and forwarding frames 104 and 96 in the respective score profiles, confirming that both request boundaries can be violated.

Evidence from the check

This executed Python harness applies the in-memory mutation, runs the existing tests, and probes generation with deterministic doubles, making the coverage comparison reproducible.

Evidence from the check

This executed driver records command, working directory, exit code, and observed output for inspection and the single before/after pair.

Command output from the check

Numbered source excerpts and search output show direct selector tests and CLI tests without sharp-option checks, locating the coverage gap.

Command output from the check

The unmodified run records 83 passes and two failures while the separate generation probe emits selected indices (12, 22), establishing the baseline.

Command output from the check

The mutated run records the same test outcomes while generation emits anchors (10, 20), showing that the existing tests do not detect the disconnection.

Command output from the check

The final git diff check exits successfully with no diff, confirming that validation left tracked sources unchanged.

Summary

Adds optional sharpness-based frame selection to VQA generation.

VQA range generation can choose sharper nearby frames.

_{Reviews (1) · Last reviewed commit: "feat(evals): add optional sharp-stride f..." · Reviewed by Greptile}

greptile-apps · 2026-10-14T14:56:45Z

+        lo = max(0, anchor - half_width)
+        hi = min(image_count - 1, anchor + half_width)


Frames escape the requested range

--sharp-stride clips its search only to the recording, not to start and the exclusive stop. For example, start=100, stop=101, stride=10 searches frames 96–104 and forwards frame 104 to generation if it has the highest sharpness. A sharper frame before start can also be selected. Users can therefore get images and questions from outside their requested range. Pass the requested bounds into the selector and clip each neighborhood to them before merging.

Artifacts

Scoped generation reproduction script

Executes real generation selection and consumes its frame iterator with fake supporting services, asserting the exact selected indices at both request boundaries.

Commands and output capture script

Runs the reproduction with sharp-stride disabled and enabled and records each command, working directory, exit code, and observed output for comparison.

Generation output with sharp-stride disabled

Captured the baseline generation runs for both score profiles; only frame 100 was loaded and forwarded, respecting the requested range.

Generation output with sharp-stride enabled

Captured generation selecting and forwarding frames 104 and 96 in the respective score profiles, confirming that both request boundaries can be violated.

_{Ran code and verified through T-Rex}

Fixed in 0a0038a: neighborhoods are clipped to [start, stop); added resolve/generate forwarding tests and CLI option coverage.

greptile-apps · 2026-10-14T14:56:45Z

+def test_select_sharp_stride_indices_picks_neighborhood_max_and_dedupes() -> None:
+    scores = {i: float(i) for i in range(30)}
+    scores[12] = 100.0
+    scores[22] = 100.0
+    selected = select_sharp_stride_indices(
+        anchors=(10, 20),
+        image_count=30,
+        half_width=4,
+        score_at=scores.__getitem__,
+    )
+    assert selected == (12, 22)


Generation path remains untested

The new tests exercise the selector and request checks, but do not verify that generate_dataset consumes the sharp-selected indices. Replacing its frame loop with the original anchors left all seven new sharp-stride cases passing and the existing test outcomes unchanged, even though generation then emitted the wrong frames. This is a non-blocking regression-coverage gap. Add a test with a fake loader and models that checks which indices reach generation, plus a CLI test that passes both new options.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Executable baseline and mutation harness

This executed Python harness applies the in-memory mutation, runs the existing tests, and probes generation with deterministic doubles, making the coverage comparison reproducible.

Executed commands and evidence capture script

This executed driver records command, working directory, exit code, and observed output for inspection and the single before/after pair.

Scoped selector, generation, and CLI test inspection

Numbered source excerpts and search output show direct selector tests and CLI tests without sharp-option checks, locating the coverage gap.

Tests and generation probe before mutation

The unmodified run records 83 passes and two failures while the separate generation probe emits selected indices (12, 22), establishing the baseline.

Tests and generation probe after disconnecting selected indices

The mutated run records the same test outcomes while generation emits anchors (10, 20), showing that the existing tests do not detect the disconnection.

Tracked source status after validation

The final git diff check exits successfully with no diff, confirming that validation left tracked sources unchanged.

_{Ran code and verified through T-Rex}

Fixed in 0a0038a: neighborhoods are clipped to [start, stop); added resolve/generate forwarding tests and CLI option coverage.

Keep sharpness search inside the requested generation range, and cover the generate/CLI path so selected indices are asserted end-to-end.

Drop custom sharp-stride neighborhoods; reuse Image.sharpness + QualityWindow (and optional brightness floor) the same way loop-closure / semantic peaks do.

moshiii-wei · 2026-10-14T17:17:11Z

Reworked per review: dropped the custom sharp-stride / k-neighborhood path. Range mode now uses mem `QualityWindow(lambda img: img.sharpness, window=...)` (optional `--min-brightness` first), matching the loop-closure / semantic-peak docs.

moshiii-wei · 2026-10-14T17:53:10Z

Added a Comparison experiment section to the PR description with the go2_short A/B:

Selection: QualityWindow sharper on 84% of nearest-in-time pairs; mean sharpness 0.796 → 0.834
Case yield (12 frames/arm, image-only families, real author+Moondream path): answered 23 → 33, reject rate 30% → 11%

Restore Sequence import in generate.py and cast QualityWindow outputs so lint mypy passes.

moshiii-wei · 2026-10-14T18:03:50Z

Fixed CI mypy failures: restored Sequence import in generate.py and cast QualityWindow outputs in select_quality_indices.

feat(evals): add optional sharp-stride frame selection for VQA generate

a4e8697

Keep uniform start/stop/stride anchors, then pick the sharpest nearby color_image in a stride-aware neighborhood so motion-blurred samples are less likely to enter deterministic VQA cases.

moshiii-wei requested review from Dreamsorcerer, TomCC7, aclauer, leshy, mustafab0, paul-nechifor and spomichter as code owners October 14, 2026 14:48

github-actions Bot reviewed Oct 14, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 14, 2026

View reviewed changes

Moshi Wei and others added 3 commits October 14, 2026 11:20

fix(evals): clip sharp-stride neighborhoods to [start, stop)

0a0038a

Keep sharpness search inside the requested generation range, and cover the generate/CLI path so selected indices are asserted end-to-end.

[autofix.ci] apply automated fixes

d7cd7b8

refactor(evals): select VQA frames with mem QualityWindow

7fff1b3

Drop custom sharp-stride neighborhoods; reuse Image.sharpness + QualityWindow (and optional brightness floor) the same way loop-closure / semantic peaks do.

moshiii-wei changed the title ~~feat(evals): optional sharp-stride frame selection for VQA generate~~ feat(evals): optional QualityWindow frame selection for VQA generate Oct 14, 2026

[autofix.ci] apply automated fixes

656aa81

fix(evals): satisfy mypy for QualityWindow VQA selection

77eb107

Restore Sequence import in generate.py and cast QualityWindow outputs so lint mypy passes.

[autofix.ci] apply automated fixes

ce0d609

Navigation Menu

Repository navigation

feat(evals): optional QualityWindow frame selection for VQA generate - #4566

feat(evals): optional QualityWindow frame selection for VQA generate#4566
moshiii-wei wants to merge 7 commits into
mainfrom
moshi/feat/vqa-sharp-stride

moshiii-wei commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment •

edited

Loading

Uh oh!

codecov Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Oct 14, 2026

Uh oh!

moshiii-wei Oct 14, 2026

Uh oh!

greptile-apps Bot Oct 14, 2026

Uh oh!

moshiii-wei Oct 14, 2026

Uh oh!

moshiii-wei commented Oct 14, 2026

Uh oh!

moshiii-wei commented Oct 14, 2026

Uh oh!

moshiii-wei commented Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		lo = max(0, anchor - half_width)
		hi = min(image_count - 1, anchor + half_width)

Conversation

moshiii-wei commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this feature?

Why do we need this?

Comparison experiment (go2_short)

Frame selection

Eval-case yield (image-only families)

How to Test

Which issue(s) does this PR close?

Checklist

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

go2 replay realtime (arm64)

Uh oh!

codecov Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

greptile-apps Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

T-Rex evidence

Summary

Uh oh!

greptile-apps Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

moshiii-wei Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

moshiii-wei Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

moshiii-wei commented Oct 14, 2026

Uh oh!

moshiii-wei commented Oct 14, 2026

Uh oh!

moshiii-wei commented Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

moshiii-wei commented Oct 14, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

codecov Bot commented Oct 14, 2026 •

edited

Loading

greptile-apps Bot commented Oct 14, 2026 •

edited

Loading