Repository navigation
Conversation
- tracking.rs: peaks followed as tracks (greedy nearest within a gate, smoothed position, confirmed after 3 maps, dropped after 5 misses); dots show confirmed tracks only, [tracking] config section - beam_signal.rs: delay and sum toward each track per STFT frame, overlap-add to a time signal, FIR decimation to 16 kHz for the speech classifier - Pipeline finds peaks and updates tracks; on_map gets the tracks with their audio since the previous map - CLI: track_NNN.wav per confirmed track (hound), `mic [maps]` - Stft::scale is used now
- speech.rs (app): voice-activity-detection per track (preprocessor and hysteresis per track id), [speech] config section; model copied to model/speech_detector.json - Classifier as git dependency over SSH, pinned; .cargo/config.toml lets cargo use the git CLI for it - GUI: green dots and speech probability in the label; checkbox renamed to "Tracks"; speech threshold slider, applied without restarting the pipeline - CLI: green dots in PNGs, speech start/stop printed - Test fixture: 4.7 s synthesized speech (16 kHz) - CLAUDE.md: plan for a longer speech hysteresis (flag bounces now)
- Beam signal: keep the common propagation delay (remove only each
mic's delay relative to the mean distance). Removing the whole
delay (~280 samples at 2 m) wrapped the window around the frame and
broke the overlap-add: a 94 Hz buzz in the track audio. Test with
off-bin band noise (a bin-centered sine hid the error).
- Speech: gain_db (+36 dB) before the classifier; the UMA-16 gives
speech at about -60 dBFS, below the -55 dB level gate, so every
frame was gated.
- Hysteresis: on_threshold (default 0.95), 500 ms hold re-armed only
by frames whose own probability reaches the threshold, 80 ms
minimum onset against snaps and claps, 50 ms smoothing; speech ends
when the track moves more than max_shift_cells.
- Echo check: a speech track that correlates with another speech
track having echo_margin_db more speech energy (p * rms^2) is a
reflection, ghost or leakage and stays blue ("echo of #N").
- GUI: markers scale with the image; hollow blue ring vs. large green
dot with halo; labels on pills.
- CLEAN-SC is the default beamformer.
- Logo in assets/ (SVG, PNGs), used as the GUI window icon and in the
README.
- Source check from the covariance, beamformer-independent
(coherent_and_received_power / source_fraction): delay-and-sum
with diagonal removal, normalized so a lone source gives its mean
mic power. The pipeline learns the noise floor (lowest subgrid
maximum of the coherent power over the last minute, so a pause
between words is enough) and gives each track its level above it
and its coherent fraction above 2 kHz.
- Tracks are shown only peaks.min_above_floor_db above the floor
("Above floor" slider, applies immediately); speech only while the
upper-band fraction reaches speech.min_source_db (leaked speech
from a talker elsewhere stays blue). "Speech" checkbox turns the
classifier off entirely.
- Tracker hits count cumulatively (a source at the gate limit never
got three in a row); position smoothing 1.0, so the dot sits on the
current peak (0.5 put it between a source and its reflection).
- CLEAN-SC: clean_equal_bins (default on), each bin's clean map
divided by its maximum before summing; otherwise the loudest, low
bins decided and marked the ceiling reflection of a source.
- GUI: one-row top bar, band as one slider with two handles, camera
choice and the equal-bins checkboxes in a settings menu, overlay
range down to 0.1 dB, labels back to "#id level %" with details on
hover.
- voice-activity-detection over HTTPS (public now), cargo git-fetch
config removed.
- A hand snapping next to a talker is inside the beam toward the talker at every speech frequency, so both beam signals are the same audio and no audio test can tell the tracks apart. A second covariance without transient frames (energy 10 dB above the median of the last 24) gives a steady map; only tracks with a peak in it can be speech. skip_transients (default off, settings menu) leaves transients out of the map itself. - The upper-band speech gate measures a track's share of the power that the other tracks do not explain: as a share of all received power, snapping raised the total by 10 dB and pushed the talker below the threshold. - peaks.range_db 6 -> 12 dB: a talker next to snapping is about 10 dB below the map maximum and lost its peak. - Hover text marks tracks that are not steady as impulsive.
- macOS camera: skip formats below 25 fps, set output size and preset, print format - GUI: settings behind a faint button over the image, saved to sound2view-gui.toml - Drop peaks on the image border, penalize sources toward the edges - Coherence threshold and windowed confirmation against noise sources
setSessionPreset(AVCaptureSessionPresetInputPriority) throws an Objective-C exception, which Rust cannot catch, so the process aborted right after printing the chosen format. setActiveFormat switches the session to that preset by itself.
The scan grid gets a margin of grid_margin_deg (15) beyond the field of view on every side, capped at 55 degrees off axis. A source outside the image then peaks on a margin cell instead of the image border, and CLEAN-SC takes it out of the visible part. Only the visible cells are handed to the map handler, peaks and tracks; peaks outside still count for the minimum distance so they suppress their leakage near the border.
With the scan grid margin, the grid border lies far outside the image, so a maximum there is better kept: it suppresses its leakage toward the image for min_dist_cells. Outside sources no longer light up the image border, so a mild center preference is enough.
JuliDi
force-pushed
the
julian/source-tracks
branch
from
November 7, 2026 10:04
747ca8b to
b31cc1f
Compare
Hand-built R gives P = 1 at the source with the removal normalization, sigma^2 I gives 0 with removal and sigma^2 / sum(g) without, and with 110 dB mic noise the stream estimate stays within 5 % of the oracle and closer to the truth than without removal. CI passes again.
Iterator zips instead of index loops in the STFT stream, the covariance test and the reference covariance, a plain negation in the mic array test, and as_chunks for the BGRA to RGBA copy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.