Performance plan: one engine that transcribes while you speak, flat with length

[dinooo13]

Eight ranked changes to the release-to-paste path, worked through in order. They came from a written plan that was executed and then removed; this description is the record of what shipped and where it differed. docs/ARCHITECTURE-PLAN.md and docs/ARCHITECTURE-REVIEW.md go too, and docs/PERFORMANCE.md arrives to describe the design that resulted.

Steps 1 to 4 take fixed costs off the release-to-paste path for every dictation. Steps 6 to 8 attack the one cost that grew with the length of the recording: the engine's window passes, all of which used to run at release. The result is that the wait is now flat. A ten minute recording pastes in about the same time as a ten second one, and the text is byte-identical to what transcribing the whole recording at release would have produced.

Pladder now has one engine. The batch engine and an intermediate sliding-window engine are both deleted; see ## One engine below.

Steps 1 to 4 — get everything else off the path

Step 1, warm the Neural Engine while the key is held. DictationCoordinator.hotkeyPressed starts a detached loop that transcribes half a second of silence immediately and then every warmupInterval, two seconds by default, until release. Every utterance is padded to the model's fixed 15 s window, so each warm-up is exactly the encoder pass the real call will make, spent while the user is still speaking. It began as a single pass at key-down; real dictations showed that is not enough for a long recording, and ## Real dictations has the numbers. Three tests: theEngineKeepsWarmingWhileTheKeyIsHeld, keyDownWarmsTheEngine, releaseDuringWarmupStillInserts.

Step 2, AVAudioEngine.pause() off the path. AVAudioEngineCapture.stop() removes the tap, drains the samples and returns; the pause() that stops CoreAudio's hardware I/O cycle now runs afterwards via pauseIfIdle(), which skips it if a new recording started meanwhile. The samples are complete before the pause begins, so nothing is lost.

Step 3, the 10 ms sleep before Cmd+V. PasteboardOutput.propagationDelay becomes an initialiser parameter defaulting to .zero, and the sleep is skipped entirely at zero. The pasteboard write is synchronous with the pasteboard server, so it has landed when the call returns, and the key event still has to cross the window server after that. The sleep guarded a race the ordering already prevents.

Step 4, clipboard snapshot at key-down. TextOutput gains prepare(), with a no-op default so other conformers keep compiling. PasteboardOutput.prepare() captures the snapshot and NSPasteboard.general.changeCount; insert uses it only when the change count still matches, so anything copied during the recording still wins. maximumItemBytes goes from 4 MB to 64 MB, since the cap existed only because the read sat on the path.

Step 5 is diagnostic only, no code. It needs overlay A/B measurements from real dictations; see ## Not done.

Steps 3 and 4 together are visible in the measurements below as a 2 ms paste.

Steps 6 to 8 — transcribe while the user speaks

Step 7 turns seam-gap repair off in the engine config.

Step 6 added StreamingTranscriptionEngine (beginUtterance, feed, endUtterance, abandonUtterance, warmPass) and AudioCapture.drain(), which hands out everything captured since the last drain and keeps recording, so stop() returns only the tail. The coordinator runs a feed task that drains once a second, tracks fedSampleCount so the minimum-duration check sees the whole utterance, and cancels and abandons on cancel, suspend and short taps. That protocol and plumbing is what step 8 runs on.

Step 8 is the engine. Windows in the batch layout do not depend on each other: each starts from a fresh decoder state, and a window's start is chosen from audio that ends before the previous window does. So every window but the last can run while the recording is still going, leaving one window and the merge at release. That is one pass at any length, and because it is the same windows in the same order through the same merge, the text is the batch engine's text rather than an approximation of it.

The window scheduling lives in a FluidAudio fork as a new IncrementalChunkProcessor, because ChunkProcessor, transcribeChunk and mergeChunks are all internal.

One engine

Step 6's sliding-window engine reached the same flat latency a different way, by carrying decoder state across windows and cutting on a fixed clock. It cost 1 to 6 points of word error rate at seams, growing with length. Step 8 gets the same latency at no accuracy cost, so it supersedes it.

With that settled, the measurements no longer justify keeping either of the other two engines, and both are deleted. Transcribing while speaking is now the only path and the default. Below 15 s there is only one window either way and the same code runs, so nothing is given up on short dictations.

An engine ID left behind by the removal resolves to the first registered engine, which EngineRegistry.make already did; AppModel now also rewrites the stored ID so the settings picker shows what is running.

The CLI loses --engine and gains --paced, since there is one engine and two ways into it. The identity gate survives intact: a paced run is compared against the same engine handed the whole buffer, which is the call it makes at release anyway.

The fork

Package.swift points at https://github.com/dinooo13/FluidAudio.git, branch incremental-chunks. The repository is public, so CI needs no credentials. This must go back to the upstream release line once the change is accepted there; a branch dependency is a temporary state.

The branch carries IncrementalChunkProcessor, a refactor that makes ChunkProcessor's per-window plan and merge tail reusable instead of inlined, and upstream main merged in. Two of the merged commits matter here:

• fix(asr/tdt): default v3 long-form to the no-mel path (silence-aligned starts) (#869) changes which chunk layout v3 uses by default, from a fixed grid to silence-aligned starts. That is the layout the incremental path had never exercised, because it was unreachable before the merge, and where its start decision could diverge from a batch run: the silence search reads forward of the nominal start, and deciding as soon as audio passed that start searched a truncated signal. The fork fixes it in the same pass by waiting until the audio the search reads has arrived, which costs nothing because the window cannot run until far more audio exists.
• The seam work in #897 and its follow-ups improves accuracy outright, which shows up below.

The change offered upstream is deliberately small: one new file plus a refactor that only moves existing code into reusable pieces, with identity tests proving the refactor changed no behaviour. It touches 172 lines of ChunkProcessor.swift.

Benchmark

Apple M1, macOS 26.6.2, one session, load average 2.12.

swift run -c release pladder-cli bench bench/fixtures --paced --all --runs 3 --pause 10

Each fixture is pushed in one-second chunks paced at real time, and only endUtterance is timed, which is what remains on the release-to-paste path. The other column is the same samples through the same engine handed the whole buffer, clock-timed, after the same idle pause.

Fixture	Audio	At release	While speaking	Identical text	WER
10s	9.7 s	0.206 s	0.257 s	yes	0.0 %
30s	31.8 s	0.416 s	0.252 s	yes	0.0 %
60s	60.8 s	0.520 s	0.278 s	yes	1.4 %
2m	125.1 s	0.941 s	0.254 s	yes	0.9 %
5m	315.6 s	2.007 s	0.275 s	yes	0.6 %
10m	631.4 s	3.665 s	0.323 s	yes	0.6 %

The identity column is the headline, not the times. Byte-identical text on all six fixtures, compared before any processor runs, with the word error rate matching on every one. The latency became flat without costing any accuracy, which is exactly what the sliding-window approach could not do.

The 10 s row is not a regression. Below one model window there are no windows to run early, so both columns execute the same code. The at-release column is one run per fixture, not a median, and an earlier run of the same comparison had the sign reversed.

Every number above is a cold release. Measured separately on the 10 s fixture, the identical path costs 0.151 s warm and 0.265 s cold. The paced bench never warms, so real dictations are faster than this table.

This table predates the upstream merge and is stale on accuracy, in the right direction. Re-measured afterwards, the 60 s fixture drops from 1.4 % to 0.0 % with no change in time (0.279 s paced, 0.552 s whole, identical). Only the 10 s and 60 s fixtures were re-run; a full six-fixture pass takes an hour of paced audio and is left for a quiet machine.

A bench fix, and a misleading first table

The paced bench's comparison column used to report FluidAudio's internal processingTime and ran with no idle pause. That was wrong twice over: processingTime starts inside FluidAudio and misses the actor hop, the decoder-state allocation and the padding copy, and a call made straight after a paced run finds the Neural Engine warm when every release the bench models is cold. Either error alone is worth more than the difference between the two columns on short audio, where they run the same code. It is now clock-timed across the whole call and takes the same --pause.

Real dictations, and the warm pass

The first dictations confirmed the cold-release problem in the app: 68.5 s of audio pasted in 0.306 s with 0.252 s of that in the engine, which is the cold figure. A window runs roughly every 13 s, so the final window at release had been idle long enough to pay for it. Hence the repeating warm pass. After it:

Audio	Release to paste	stop	engine	process	paste
8.2 s	0.239 s	0.020	0.182	0.000	0.002
12.0 s	0.232 s	0.022	0.170	0.000	0.002
37.8 s	0.263 s	0.012	0.216	0.000	0.002
64.7 s	0.264 s	0.015	0.213	0.000	0.002

The comparable long dictation went from 0.252 s to 0.213 s in the engine and 0.306 s to 0.264 s overall. Flatness holds in the app and not just on fixtures: 37.8 s and 64.7 s cost the same, and the roughly 38 ms between the short pair and the long pair is the merge, which only multi-window recordings pay.

Two honest notes. This is one dictation per length. And the repeat cuts both ways: a CoreML call already running cannot be cancelled, so a release landing inside a warm pass waits for it, roughly one dictation in fourteen at a two second interval. None of the four hit it, but four samples cannot rule it out. The interval is settable and two seconds is a starting point, not a measured optimum.

These lines also needed a fix to the logging. The four stage timings were interpolated into the log without privacy: .public, so the system redacted them and the line read <private> where the numbers should be. That predates this branch and made the measurement procedure in CLAUDE.md impossible to follow. The stage columns above are the first time those numbers have been visible.

stop is now the second largest stage at 12 to 22 ms, almost all of it removing the audio tap, which blocks until the callback currently running finishes. Moving it off the path means draining before removing the tap, which risks the few milliseconds of audio in the in-flight callback. That is an accuracy trade rather than a free win and is not attempted here.

Verification

• swift build and swift test pass: 98 tests in 10 suites. CI green on macos-26 including the ad-hoc-signed bundle step.
• In the fork: 131 tests pass, up from a 128-test baseline, including two identity tests that feed audio to IncrementalChunkProcessor in pieces and compare against a batch transcription of the same samples, at both long and short lengths.
• Steps 1 to 4 are invisible to the CLI bench, which calls the engine directly and never touches capture, the clipboard or the paste. Their evidence is the app's own log line, above.

Not verified

• Whether the periodic warm pass holds up. Four real dictations say it works and that none caught a pass in flight, but that is one sample per length. The failure signature is engine sitting well above engine-time; if it shows up often, lengthen the interval.
• Step 3 across apps. Zero propagation delay has not been exercised against Terminal, Safari, Xcode, an Electron app and a Claude session. A stale paste fails the gate, and 2 ms is the next value to try.
• Step 2's microphone indicator, including two dictations in quick succession.
• Step 4 with a rich clipboard: that a screenshot is restored after the paste, and that copying during a recording still beats the key-down snapshot.

Not done

• A full six-fixture re-run after the upstream merge, to replace the stale accuracy column.
• Step 5's diagnostic: ten dictations with the overlay set to Menu Bar, ten with Compact and glass on, comparing the remainder after subtracting the four stages. Only if they differ by more than 5 ms does the spinner get replaced.

Out of scope

• Upstreaming the fork. The diff is kept small and rebasable so it can be offered as a pull request there; until it lands, Package.swift points at the branch.
• Taking stop off the path, for the accuracy reason above.
• The smaller-window model from the plan's last section. Nothing here goes below one padded encoder pass, and that floor needs a different model repository.

🤖 Generated with Claude Code