Repository navigation
feat: voice mode, a global voice assistant over all conversations - #234
Merged
Merged
Conversation
The main screen hands the session listing to the sidebar only when it differs from the last one it saw. Its initial value is an empty list, so an empty listing (a fresh install, or a new data directory) never reached the sidebar. The sidebar then never counted its sessions as loaded and never laid out the folders of the projects saved in projects.json: the Projects view stayed empty. Gpui now records when the first listing arrived, and the main screen hands that first listing over even when it is empty.
A model-to-deployment mapping becomes part of the AI Core provider form only when its add button is clicked. A mapping typed into the add row and saved right away was dropped without notice, and the form closes on save, so the loss showed only when the provider was opened again. Saving now includes a fully typed add row (model ID and deployment ID) unless a mapping for that model already exists.
A realtime speech model you talk to about your conversations: what is running, what an agent answered, start a conversation or send one a message. It runs next to the sessions, not inside one, and acts only through conversation tools. - llm::realtime: OpenAI Realtime protocol and WebSocket transport; connectors for OpenAI-type providers and AI Core deployments. - audio crate: microphone and speakers at 24 kHz; macOS uses the voice-processing I/O unit (echo cancellation), elsewhere cpal with half-duplex gating. - code_assistant_core::voice: VoiceService, the agent loop, the tools (list_conversations, list_projects, message_conversation, get_conversation) and notifications when a conversation finishes or waits. The pure Floor state machine keeps them from talking over the user or the model: they wait for playback to drain and a quiet period. - SessionService: session_content and session_activity_states. - GPUI: microphone, mute and status in the title bar, a transcript popover, and Settings -> Voice (voice.json). - macOS bundle: microphone usage description and audio-input entitlement. See docs/voice-mode.md.
SessionService::create_session created a session bound to a project without a model configuration, so the first agent run failed with "Session has no model configuration". Sessions without a project already got the default model; project sessions now do too.
VoiceService waited for the agent task on Stop, but the task does not read its control channel while it connects or backs off between reconnects, and no connection attempt had a timeout. On a network that swallows the handshake, voice mode stayed on "Connecting" and the worker hung, with later Start, Stop and Mute commands queued behind it. Every connection attempt now gives up after 20 seconds, and Stop aborts an agent that does not end within a second.
Late audio deltas were dropped only for the item whose playback the barge-in stopped. A response the user interrupted before its first audio arrived, for example a notification whose response.create was in flight when the user started to speak, had no such item: its audio played over the user, and nothing truncated it. Model audio now plays only while the model has the floor.
Three gaps in the floor state machine: - Cancelling our own response on its creation (the user barged in while response.create was in flight) also cancelled the user-turn timer. If no answer to the user followed, the floor stayed with the user and no notification went out until the user spoke again. - A tool output that arrived during the user's turn waited for the model's answer to the user. When the turn ended without one, the output was never answered. - With several function calls in one response, the first output to complete triggered response.create. The model answered on partial results and the next output started another response. Tool outputs now count as unanswered until a response starts. The model answers them once no tool runs any more and the floor is free: after playback, from Cooling or Idle, or when the user's turn ends without a response.
Muting dropped the microphone audio but told the floor nothing. Muted mid-sentence, the server never reported the end of speech, so the floor stayed with the user and no notification was delivered, although muted users are meant to get them. Muting now ends the user's turn like the end of speech does. A voice tool that panicked or hung never reported back. The floor counted it as running for the rest of the voice session and held every notification back. Each call now reports an error after a panic or after 30 seconds.
SessionService::session_content ran on the global lane while holding the manager lock, and the file-backed SessionSource loaded sessions with every stored tool result resolved from its blob. A voice get_conversation on a large session held up all frontend commands and the agents' checkpoint commits, for tool results it never reads. session_content now runs as an I/O query on the blocking pool, and the SessionSource loads sessions unresolved: session queries read the messages only, never the execution records, which also speeds up the search_sessions and get_session_content tools.
A voice failure went into the error popover, which belongs to the viewed session: any activity change of that session cleared the message, and dismissing it sent ClearError, which reset the session. The title bar now shows a "Voice stopped" chip with the reason in its tooltip until voice mode starts again.
The output callback runs on the audio device's real-time thread and took the same mutex that push() held while it extended one VecDeque of samples. The model sends audio faster than real time, so a long answer grew that queue to megabytes, and a reallocation (or the tokio thread being preempted) under the lock made the callback miss its deadline: clicks and dropouts mid-answer. The cpal input callback took the lock too, for is_playing(). push() now converts a chunk before it takes the lock and only moves it into the queue, render() plays one silent buffer instead of waiting when the lock is held, and the playing flag is an atomic.
On macOS a denied microphone permission does not make the voice processing unit fail: it records silence. Voice mode then connected, showed "Listening" and never heard anything, with nothing pointing to the cause. The macOS backend now checks the AVFoundation authorization status before it opens the devices and fails with a message that names the setting. An undecided permission still lets the system ask on the first capture.
The barge-in test asserted that the speakers were cleared as soon as it received the truncate event, but the agent clears them only after it sent that event, on its own task. On a busy CI runner the assertion ran first. The test also slept 50 ms and assumed the audio delta was queued by then. Tests now wait for the fake devices to reach the expected state. A 20 ms delay injected before clear_playback reproduced the CI failure and passes now; 320 runs in eight parallel loops passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds voice mode: a global voice assistant you talk to about your conversations — what is running, what an agent answered, start a conversation in a project or send one a message. It runs next to the sessions, not inside one, and acts only through conversation tools. The design is in
docs/voice-mode.md.Realtime model and audio
llm::realtime: the OpenAI Realtime protocol over WebSocket, with connectors for OpenAI-type providers and AI Core deployments.audiocrate: microphone and speakers at 24 kHz. macOS uses the voice-processing I/O unit (echo cancellation); elsewherecpalwith half-duplex gating. Builds without thevoicefeature link no audio backend.Voice agent (
code_assistant_core::voice)VoiceServiceand the agent loop, with toolslist_conversations,list_projects,message_conversationandget_conversation.Floorstate machine keeps them from talking over the user or the model: they wait until playback has drained, plus a quiet period.SessionServicegainssession_contentandsession_activity_statesfor the tools.GPUI
voice.json.Fixes
Testing
Floorstate machine, the voice tools, notifications, the voice config, the realtime connectors and protocol, and the audio resampler.cargo test,cargo clippy --all-targets --all-features -- -D warningsandcargo fmt --checkpass.