diff --git a/site/src/content/docs/user-guide/sdk/bidi/models/bedrock.mdx b/site/src/content/docs/user-guide/sdk/bidi/models/bedrock.mdx index d803aedd6d..3fa0df9d21 100644 --- a/site/src/content/docs/user-guide/sdk/bidi/models/bedrock.mdx +++ b/site/src/content/docs/user-guide/sdk/bidi/models/bedrock.mdx @@ -1,168 +1,124 @@ --- title: Bedrock Nova Sonic -description: 'Build real-time speech-to-speech agents with Amazon Bedrock Nova Sonic and Strands. Bidirectional audio streaming with barge-in handling and tool calling.' +description: "Configure Amazon Nova Sonic for BidiAgent: set AWS credentials, choose a voice, tune turn detection, and preserve context across connection restarts." +languages: [python] tags: [bidi-streaming, aws, bedrock] sourceLinks: - path: strands-py/src/strands/bidi/models/bedrock.py + - path: strands-py/src/strands/bidi/models/configs.py redirectFrom: - docs/user-guide/concepts/bidirectional-streaming/models/nova_sonic - docs/user-guide/sdk/bidirectional-streaming/models/bedrock --- +[Amazon Nova Sonic][nova-sonic] brings expressive, multilingual voice conversations +to your agents. Users can speak naturally, interrupt a reply, and ask the agent to +take action through tools. -[Amazon Nova Sonic](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html) provides real-time, conversational interactions through bidirectional audio streaming. Amazon Nova Sonic processes and responds to real-time speech as it occurs, enabling natural, human-like conversational experiences. Key capabilities and features include: - -- Adaptive speech response that dynamically adjusts delivery based on the prosody of the input speech. -- Graceful handling of user barge-ins without dropping conversational context. -- Function calling and agentic workflow support for building complex AI applications. -- Robustness to background noise for real-world deployment scenarios. -- Multilingual support with expressive voices and speaking styles. Expressive voices are offered, including both masculine-sounding and feminine sounding, in seven languages: English (US, UK, AU, IN), French, Italian, German, Spanish (US), Portuguese (BR), and Hindi. -- Recognition of varied speaking styles across all supported languages. +Follow the [quickstart](../quickstart.mdx) to build a voice agent that listens, +speaks, and calls tools. ## Installation -:::caution[Python 3.12+ Required] -Nova Sonic requires Python 3.12 or higher due to its experimental AWS SDK dependency. +:::note[Python version] +Nova Sonic requires Python 3.12 or later. ::: -Nova Sonic is included in the base bidirectional streaming dependencies for Strands Agents. - -To install it, run: +Install the Nova Sonic extra: ```bash -pip install 'strands-agents[bidi,bidi-io,bidi-pyaudio]' +pip install "strands-agents[bidi]" ``` -Or to install all bidirectional streaming providers at once: +For local audio dependencies and device setup, see +[Audio I/O](../io.mdx#audio-io). -```bash -pip install 'strands-agents[bidi-all,bidi-pyaudio]' -``` +## Credentials -## Usage +Configure AWS credentials with permission to invoke Nova Sonic in a +[supported region][nova-regions]. +`BedrockNovaSonicModel` uses Boto3's [credential chain][boto3-credentials], including +environment variables, named profiles, and IAM roles. See +[Setting up AWS credentials][credentials-setup] +for shared setup instructions. -After installing the Bedrock Nova Sonic and local audio extras, create a voice agent: +To use a named profile, pass a Boto3 session with its region: ```python -import asyncio +import boto3 -from strands.bidi.agent import BidiAgent -from strands.bidi.io import AudioIO from strands.bidi.models import BedrockNovaSonicModel -from strands.vended_tools import notebook - - -async def main() -> None: - model = BedrockNovaSonicModel( - model_id="amazon.nova-2-sonic-v1:0", - region="us-east-1", - voice="tiffany", - ) - agent = BidiAgent(model=model, tools=[notebook]) - audio_io = AudioIO() - await agent.run(inputs=[audio_io.input()], outputs=[audio_io.output()]) - - -if __name__ == "__main__": - asyncio.run(main()) -``` - -## Cross-Modal Input -Nova Sonic accepts text input at any point during an active voice session, without interrupting or waiting on the audio stream. Send [text input](../content.mdx#send-input) the same way you would outside of a live conversation: - -```python -await agent.send({"text": "What's the weather in Seattle?"}) +model = BedrockNovaSonicModel( + model_id="amazon.nova-2-sonic-v1:0", + boto_session=boto3.Session(profile_name="voice-agent", region_name="us-east-1"), +) ``` -Text and audio input can be interleaved freely. Sending text does not require pausing -the microphone or waiting for the model to finish speaking. - -## Credentials +Without a custom session, you can pass `region` directly. If omitted, the model +uses Boto3's configured region, falling back to `us-east-1`. -:::caution[Nova Sonic is available in us-east-1, us-west-2, eu-north-1, and ap-northeast-1.] -::: - -Nova Sonic requires AWS credentials for access. `BedrockNovaSonicModel` uses an experimental [Bedrock client](https://github.com/aws/aws-sdk-python/tree/develop/clients/aws-sdk-bedrock-runtime/src/aws_sdk_bedrock_runtime), which accepts credentials in the following ways: - -**Option 1: Environment Variables** - -```bash -export AWS_ACCESS_KEY_ID=your_access_key -export AWS_SECRET_ACCESS_KEY=your_secret_key -export AWS_SESSION_TOKEN=your_session_token # If using temporary credentials -export AWS_REGION=your_region_name -``` +## Configuration -**Option 2: Boto3 Session** +Create a model with the voice and audio settings for your application, then pass it +to `BidiAgent`. This example selects the `tiffany` voice and 24 kHz output audio: ```python -import boto3 +from strands.bidi.agent import BidiAgent from strands.bidi.models import BedrockNovaSonicModel - -boto_session = boto3.Session( - aws_access_key_id="your_access_key", - aws_secret_access_key="your_secret_key", - aws_session_token="your_session_token", # If using temporary credentials - region_name="your_region_name", - profile_name="your_profile" # Optional: Use a specific profile -) model = BedrockNovaSonicModel( model_id="amazon.nova-2-sonic-v1:0", - boto_session=boto_session, + region="us-east-1", + voice="tiffany", + audio={"output": {"sample_rate": 24000}}, ) +agent = BidiAgent(model=model) ``` -For more details on this approach, please refer to the [boto3 session docs](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html). - -## Configuration - -### Client Options - -| Parameter | Description | Default | -| --------- | ----------- | ------- | -| `boto_session` | A `boto3.Session` instance under which AWS credentials are configured. | `None` | -| `region` | Region under which credentials are configured. Cannot use if providing `boto_session`. | `us-east-1` | - -### Model Config - -| Parameter | Description | Example | Options | -| --------- | ----------- | ------- | ------- | -| `model_id` | Nova Sonic model identifier. | `"amazon.nova-2-sonic-v1:0"` | Nova Sonic model IDs | -| `audio` | Input and output stream options. | `{"output": {"sample_rate": 24000}}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicAudioConfig) | -| `voice` | Output voice identifier. Defaults to `"matthew"`. | `"tiffany"` | Nova Sonic voices | -| `params` | Provider-specific session parameters, such as inference and turn detection configuration. | `{"inferenceConfiguration": {"temperature": 0.7}}` | [`sessionStart` fields](https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-input-events.html) | -| `connection` | Restart timing overrides. | `{"auto_restart": False}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.ConnectionConfig) | - -:::caution[Conversation History Limits] -Nova Sonic caps conversation history passed via `messages` at 50KB per message and 200KB total. Messages exceeding the per-message limit are truncated; if the total exceeds 200KB, the oldest messages are dropped until it fits. This happens silently (only visible via debug-level logs) and only applies to history provided at connection start, not to turns generated during the live conversation. -::: - -## Connection Restart - -Nova Sonic keeps a single connection open for about 8 minutes. Before the connection reaches that limit, `BidiAgent` restarts it and replays the conversation history into the new connection, so the conversation continues without a break. The agent emits a [`BidiConnectionRestartEvent`](../events.mdx#connection-restarts) when this happens; treat it as informational rather than an error. - -See [Connection restarts](../agent.mdx#connection-restarts) for the full lifecycle. - -## Troubleshooting +Choose a voice from Nova Sonic's [voices and languages][nova-voices]. Audio uses +mono PCM, with 16 kHz input and output by default. Set either stream's sample rate +through [`audio`][audio-config]. -### Module Not Found - -If you encounter the error `ModuleNotFoundError: No module named 'aws_sdk_bedrock_runtime'`, this means the experimental Bedrock runtime dependency hasn't been properly installed in your environment. To fix this, run `pip install 'strands-agents[bidi]'`. - -:::note[Python Version Requirement] -Nova Sonic requires Python 3.12+ due to the experimental AWS SDK dependency. If you're using an older Python version, you'll need to upgrade. -::: +See the [`BedrockNovaSonicModel` API reference][model-api] for all constructor +options. -### Hanging +### Session settings -When credentials are misconfigured, the model provider does not throw an exception (a quirk of the underlying experimental Bedrock client). As a result, the provider allows the user to proceed forward with a call to `receive`, which emits no events and thus presents an indefinite hanging behavior. +Pass Nova Sonic's [session settings][nova-input-events] through `params` to tune +inference and turn detection. For example, `LOW` endpointing sensitivity allows +longer pauses before the model decides the user has finished speaking: -As a reminder, Nova Sonic is only available in us-east-1, us-west-2, eu-north-1, and ap-northeast-1. +```python +from strands.bidi.models import BedrockNovaSonicModel -## References +model = BedrockNovaSonicModel( + model_id="amazon.nova-2-sonic-v1:0", + params={"turnDetectionConfiguration": {"endpointingSensitivity": "LOW"}}, +) +``` -- [Nova Sonic](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html) -- [Experimental Bedrock Client](https://github.com/aws/aws-sdk-python/tree/develop/clients/aws-sdk-bedrock-runtime) -- [Python API Reference](@api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicModel) +Use the provider's field names inside `params`, as shown above. + +## Connection restarts + +Nova Sonic limits each connection to eight minutes. By default, `BidiAgent` +schedules a restart after seven minutes and replays conversation history into the +new connection. See [Connection restarts](../agent.mdx#connection-restarts) to +customize the timing and track the transition. + +On startup and restart, the adapter replays text from the +[conversation history][nova-history], +including [completed transcripts](../events.mdx#messages). It truncates individual +messages to 50 KiB and retains up to 200 KiB of recent history. These limits apply +to the history sent when opening a connection. + +[nova-sonic]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html +[nova-regions]: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-amazon-nova-2-sonic.html +[nova-voices]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-language-support.html +[nova-input-events]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-input-events.html +[nova-history]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-chat-history.html +[boto3-credentials]: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html +[credentials-setup]: ../../model-providers/amazon-bedrock.mdx#setting-up-aws-credentials +[audio-config]: @api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicAudioConfig +[model-api]: @api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicModel.__init__ diff --git a/site/src/content/docs/user-guide/sdk/bidi/models/google.mdx b/site/src/content/docs/user-guide/sdk/bidi/models/google.mdx index e3cb33894b..46d51b7b27 100644 --- a/site/src/content/docs/user-guide/sdk/bidi/models/google.mdx +++ b/site/src/content/docs/user-guide/sdk/bidi/models/google.mdx @@ -1,127 +1,114 @@ --- title: Google Gemini Live -description: "Build real-time voice agents with Google's Gemini Live API and Strands. Stream audio and text over WebSocket with barge-ins and tool calling." +description: "Configure Google Gemini Live for BidiAgent: set credentials, choose a voice and audio settings, tune turn detection, and manage connection restarts." +languages: [python] tags: [bidi-streaming] sourceLinks: - path: strands-py/src/strands/bidi/models/google.py + - path: strands-py/src/strands/bidi/models/configs.py redirectFrom: - docs/user-guide/concepts/bidirectional-streaming/models/gemini_live - docs/user-guide/sdk/bidirectional-streaming/models/google --- +[Google Gemini Live][gemini-live] brings multilingual voice conversations to your +agents, with support for text and images alongside speech. Users can share context, +interrupt a reply, and ask the agent to take action through tools. -The [Gemini Live API](https://ai.google.dev/gemini-api/docs/live) lets developers create natural conversations by enabling a two-way WebSocket connection with the Gemini models. The Live API processes data streams in real time. Users can barge in with new input while the model is responding, similar to a real conversation. Key features include: - -- **Multimodal Streaming**: The API supports streaming of text, audio, and video data. -- **Bidirectional Interaction**: The user and the model can provide input and output at the same time. -- **Barge-in**: Users can speak during the model's response, and the model adjusts its response. -- **Tool Use and Function Calling**: The API can use external tools to perform actions and get context while maintaining a real-time connection. -- **Provider Sessions**: Maintains conversation context within a Gemini Live session. For SDK session persistence, see [Session Management](../session-management.mdx). -- **Secure Authentication**: Uses tokens for secure client-side authentication. +Follow the [quickstart](../quickstart.mdx) to build a voice agent that listens, +speaks, and calls tools. ## Installation -The Google Gemini Live provider is configured as an optional dependency in Strands Agents. - -To install it, run: +Install the Google Gemini Live extra: ```bash -pip install 'strands-agents[bidi-google,bidi-io,bidi-pyaudio]' +pip install "strands-agents[bidi-google]" ``` -Or to install all bidirectional streaming providers at once: +For local audio dependencies and device setup, see +[Audio I/O](../io.mdx#audio-io). + +## Credentials + +Create an API key in [Google AI Studio][api-keys] and set it in your environment: ```bash -pip install 'strands-agents[bidi-all,bidi-pyaudio]' +export GOOGLE_API_KEY="your-api-key" ``` -## Usage +`GoogleGeminiLiveModel` reads `GOOGLE_API_KEY` automatically. You can also pass the +key through `client_args={"api_key": "your-api-key"}`. For other client options, +see the [Google GenAI client reference][client-api]. -After installing the Gemini Live and local audio extras, create a voice agent: +## Configuration -```python -import asyncio +Configure the Gemini Live model, output voice, and input audio, then pass the model +to `BidiAgent`: +```python from strands.bidi.agent import BidiAgent -from strands.bidi.io import AudioIO from strands.bidi.models import GoogleGeminiLiveModel -from strands.vended_tools import notebook - -async def main() -> None: - model = GoogleGeminiLiveModel( - model_id="gemini-3.8-live", - voice="Kore", - client_args={"api_key": ""}, - ) - agent = BidiAgent(model=model, tools=[notebook]) - - audio_io = AudioIO() - await agent.run(inputs=[audio_io.input()], outputs=[audio_io.output()]) - - -if __name__ == "__main__": - asyncio.run(main()) +model = GoogleGeminiLiveModel( + model_id="gemini-3.8-live", + voice="Kore", + audio={"input": {"sample_rate": 48000}}, +) +agent = BidiAgent(model=model) ``` -## Configuration - -### Client Options - -Pass Google GenAI client options through `client_args`. For the supported fields, see -the [Google GenAI client reference](https://googleapis.github.io/python-genai/genai.html#genai.client.Client). - -### Model Config +Choose a voice from Google's [voice options][voices]. Audio uses mono PCM, with +16 kHz input by default and fixed 24 kHz output. Set `audio.input.sample_rate` to +match your input source; Gemini resamples incoming audio as needed. The adapter +enables transcripts for both user speech and model responses. -| Parameter | Description | Example | Options | -| --------- | ----------- | ------- | ------- | -| `model_id` | Gemini Live model identifier. | `"gemini-3.8-live"` | [Gemini models](https://ai.google.dev/gemini-api/docs/models) | -| `audio` | Input audio options. | `{"input": {"sample_rate": 48000}}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.GoogleGeminiLiveAudioConfig) | -| `voice` | Prebuilt output voice name. Uses the provider default when omitted. | `"Kore"` | [Voices and languages](https://docs.cloud.google.com/text-to-speech/docs/list-voices-and-types) | -| `params` | Gemini Live session parameters. | `{"temperature": 0.7}` | [`LiveConnectConfig`](https://googleapis.github.io/python-genai/genai.html#genai.types.LiveConnectConfig) | -| `connection` | Restart timing overrides. | `{"auto_restart": False}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.ConnectionConfig) | +See the [`GoogleGeminiLiveModel` API reference][model-api] for all constructor options. -### Additional Provider Options +### Session settings -Use direct options such as `voice` for common settings. For additional Google GenAI -options, pass `params` using snake_case field names. +Pass Google's [`LiveConnectConfig` settings][session-settings] through `params` to +customize the conversation. Gemini uses [automatic voice activity detection][vad] +by default. This example allows 800 milliseconds of silence before detecting the +end of speech: ```python from strands.bidi.models import GoogleGeminiLiveModel model = GoogleGeminiLiveModel( model_id="gemini-3.8-live", - client_args={"api_key": ""}, - voice="Kore", - params={"temperature": 0.7, "speech_config": {"language_code": "en-US"}}, + params={ + "realtime_input_config": { + "automatic_activity_detection": {"silence_duration_ms": 800}, + }, + }, ) ``` -Nested dictionaries in `params` merge with the existing configuration, preserving -unspecified fields. If a setting overlaps with a default or direct option, `params` -takes precedence. - -Calling `update_config(params=...)` replaces the entire `params` dictionary. The new -values take effect on the next `start()` or `restart()`. - -## Session Resumption - -For [connection restarts](../agent.mdx#connection-restarts), the provider resumes the same server-side session through Gemini's [session resumption](https://ai.google.dev/gemini-api/docs/live-session) handle, carrying context across the restart without replaying history. Resumed sessions persist up to 24 hours; after that, create a new `GoogleGeminiLiveModel` instance to continue the conversation. - -## Troubleshooting - -### Module Not Found - -If you encounter the error `ModuleNotFoundError: No module named 'google.genai'`, this means the `google-genai` dependency hasn't been properly installed in your environment. To fix this, run `pip install 'strands-agents[bidi-google]'`. - -### API Key Issues - -Set your Google AI API key through `client_args` or the `GOOGLE_API_KEY` -environment variable. You can obtain an API key from -[Google AI Studio](https://aistudio.google.com/app/apikey). - -## References - -- [Gemini Live API](https://ai.google.dev/gemini-api/docs/live) -- [Gemini API Reference](https://googleapis.github.io/python-genai/genai.html#) -- [Python API Reference](@api/python/strands.bidi.models#strands.bidi.models.GoogleGeminiLiveModel) +Use the Google GenAI SDK's snake_case field names inside `params`. Nested settings +merge with the defaults, and `params` takes precedence over direct options such as +`voice`. + +## Connection restarts + +Gemini limits each connection to [about 10 minutes][session-lifetime]. By default, +`BidiAgent` schedules a restart after nine minutes. The adapter uses +[session resumption][session-resumption] to reconnect to the same server-side +conversation. If resumption is unavailable or fails, it opens a new session and +replays text from conversation history, including completed transcripts. + +The adapter also enables [context window compression][context-compression] by +default, allowing the session to continue as its history grows. Set +`connection.restart_after_s` to customize the restart schedule. See +[Connection restarts](../agent.mdx#connection-restarts) for timing and events. + +[gemini-live]: https://ai.google.dev/gemini-api/docs/live +[api-keys]: https://aistudio.google.com/app/apikey +[client-api]: https://googleapis.github.io/python-genai/genai.html#genai.client.Client +[voices]: https://ai.google.dev/gemini-api/docs/live-guide#change-voice-and-language +[session-settings]: https://googleapis.github.io/python-genai/genai.html#genai.types.LiveConnectConfig +[vad]: https://ai.google.dev/gemini-api/docs/live-guide#configure-automatic-vad +[session-lifetime]: https://ai.google.dev/gemini-api/docs/live-session#maximum-session-duration +[session-resumption]: https://ai.google.dev/gemini-api/docs/live-session#session-resumption +[context-compression]: https://ai.google.dev/gemini-api/docs/live-session#context-window-compression +[model-api]: @api/python/strands.bidi.models#strands.bidi.models.GoogleGeminiLiveModel.__init__ diff --git a/site/src/content/docs/user-guide/sdk/bidi/models/index.mdx b/site/src/content/docs/user-guide/sdk/bidi/models/index.mdx index 0fc5ff7232..d87eb12a0c 100644 --- a/site/src/content/docs/user-guide/sdk/bidi/models/index.mdx +++ b/site/src/content/docs/user-guide/sdk/bidi/models/index.mdx @@ -61,11 +61,15 @@ the connection opens, not to the connection already in progress. ## Audio -Realtime models take raw audio in and send raw audio back: mono PCM, at a sample rate -the model defines. The model owns the audio format, so microphones, speakers, and -client applications adapt to it rather than the other way around. Each provider page -covers the audio options its model supports, and [I/O Streams](../io.mdx) covers how -audio devices and custom streams match the model's format. +Use the model's audio configuration to set up capture and playback. The built-in +providers implement [`AudioCapable`][audio-capable], which exposes +`get_audio_config()`. It returns separate `input` and `output` settings containing +each stream's sample rate, channel count, and encoding. + +[`AudioIO`](../io.mdx#audio-io) reads these settings automatically. Custom +[I/O streams](../io.mdx#custom-io) can read them to configure audio capture and +playback. The built-in providers use mono PCM; sample rates and configuration +options vary by provider. See each provider's page for its supported settings. ## Connection limits @@ -78,8 +82,8 @@ restart in one of two ways: - **History replay**: Nova Sonic and OpenAI Realtime open a fresh connection and receive the conversation history. - **Session resumption**: Gemini Live reconnects to the same server-side session - through [session resumption](https://ai.google.dev/gemini-api/docs/live-session), so - no history is resent. + through [session resumption](google.mdx#connection-restarts), falling back to + history replay if resumption is unavailable. Each provider sets its own default restart timing, and the `connection` option overrides it or turns automatic restarts off. For the restart sequence and the events @@ -93,3 +97,5 @@ yields Strands events as the model responds, in the order described in [Event ordering](../events.mdx#event-ordering). See the [`BidiModel` API reference](@api/python/strands.bidi.models#strands.bidi.models.BidiModel) for the contract, and the built-in providers in `strands/bidi/models/` for working examples. + +[audio-capable]: @api/python/strands.bidi.models#strands.bidi.models.AudioCapable diff --git a/site/src/content/docs/user-guide/sdk/bidi/models/openai.mdx b/site/src/content/docs/user-guide/sdk/bidi/models/openai.mdx index b3e10f37b1..d56100d0d8 100644 --- a/site/src/content/docs/user-guide/sdk/bidi/models/openai.mdx +++ b/site/src/content/docs/user-guide/sdk/bidi/models/openai.mdx @@ -1,93 +1,78 @@ --- title: OpenAI Realtime -description: 'Build low-latency voice agents with the OpenAI Realtime API and Strands. Configure speech-to-speech streaming, barge-ins, and tool calling.' +description: "Configure OpenAI Realtime for BidiAgent: set credentials, choose a voice and transcription model, tune turn detection, and manage connection restarts." +languages: [python] tags: [bidi-streaming] sourceLinks: - path: strands-py/src/strands/bidi/models/openai.py + - path: strands-py/src/strands/bidi/models/configs.py redirectFrom: - docs/user-guide/concepts/bidirectional-streaming/models/openai_realtime - docs/user-guide/sdk/bidirectional-streaming/models/openai --- +[OpenAI Realtime][realtime] brings responsive voice conversations to your agents, +with support for text and images alongside speech. Users can share context, +interrupt a reply, and ask the agent to take action through tools. -The [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime) is a speech-to-speech interface that enables low-latency, natural voice conversations with AI. Key features include: - -- **Bidirectional Interaction**: The user and the model can provide input and output at the same time. -- **Barge-in**: Users can speak while the model is responding, like in human conversations. -- **Multimodal Streaming**: The API supports streaming of text and audio data. -- **Tool Use and Function Calling**: Can use external tools to perform actions and get context while maintaining a real-time connection. -- **Secure Authentication**: Uses tokens for secure client-side authentication. +Follow the [quickstart](../quickstart.mdx) to build a voice agent that listens, +speaks, and calls tools. ## Installation -OpenAI Realtime is configured as an optional dependency in Strands Agents. - -To install it, run: +Install the OpenAI Realtime extra: ```bash -pip install 'strands-agents[bidi-io,bidi-openai,bidi-pyaudio]' +pip install "strands-agents[bidi-openai]" ``` -Or to install all bidirectional streaming providers at once: +For local audio dependencies and device setup, see +[Audio I/O](../io.mdx#audio-io). + +## Credentials + +Create an [OpenAI API key][api-keys] and set it in your environment: ```bash -pip install 'strands-agents[bidi-all,bidi-pyaudio]' +export OPENAI_API_KEY="your-api-key" ``` -## Usage +`OpenAIRealtimeModel` reads `OPENAI_API_KEY` automatically. You can also pass the key +directly through `api_key`. For organization and project options, see the +[constructor reference][model-api]. -After installing the OpenAI Realtime and local audio extras, create a voice agent: +## Configuration -```python -import asyncio +Configure the Realtime model, user transcription, and output voice, then pass the +model to `BidiAgent`: +```python from strands.bidi.agent import BidiAgent -from strands.bidi.io import AudioIO from strands.bidi.models import OpenAIRealtimeModel -from strands.vended_tools import notebook - -async def main() -> None: - model = OpenAIRealtimeModel( - transcription_model_id="gpt-transcribe", - model_id="gpt-realtime-2.1", - voice="coral", - api_key="", - ) - agent = BidiAgent(model=model, tools=[notebook]) - - audio_io = AudioIO() - await agent.run(inputs=[audio_io.input()], outputs=[audio_io.output()]) - - -if __name__ == "__main__": - asyncio.run(main()) +model = OpenAIRealtimeModel( + model_id="gpt-realtime-2.1", + transcription_model_id="gpt-transcribe", + voice="coral", +) +agent = BidiAgent(model=model) ``` -## Configuration - -### Client Options +The `transcription_model_id` argument is required: choose a +[transcription model][transcription-model] for user speech transcripts, or pass +`None` to disable them. The Realtime model processes audio directly, independently +of this transcription. -| Parameter | Description | Example | Options | -| --------- | ----------- | ------- | ------- | -| `api_key` | OpenAI API key used for authentication | `sk-...` | [reference](https://platform.openai.com/docs/api-reference/authentication) | -| `organization` | Organization associated with the connection. Used for authentication if required. | `myorg` | [reference](https://platform.openai.com/docs/api-reference/authentication) -| `project` | Project associated with the connection. Used for authentication if required. | `myproj` | [reference](https://platform.openai.com/docs/api-reference/authentication) -| `timeout_s` | OpenAI documents a 60 minute limit on realtime sessions ([docs](https://platform.openai.com/docs/guides/realtime-conversations#session-lifecycle-events)). However, OpenAI does not emit any warnings when approaching the limit. As a workaround, we allow users to configure a timeout (in seconds) on the client side to gracefully handle the connection closure. | `3000` | `[1, 3000]` (in seconds) +Choose a voice from OpenAI's [voice options][voices]. This adapter requires mono +PCM at 24 kHz for both audio input and output. -### Model Config +See the [`OpenAIRealtimeModel` API reference][model-api] for all constructor options. -| Parameter | Description | Example | Options | -| --------- | ----------- | ------- | ------- | -| `model_id` | OpenAI Realtime model identifier. | `"gpt-realtime-2.1"` | [OpenAI models](https://platform.openai.com/docs/models) | -| `transcription_model_id` | Required input transcription model identifier. Pass `None` to disable user transcription. | `"gpt-transcribe"` | [GPT-Transcribe](https://developers.openai.com/api/docs/models/gpt-transcribe) | -| `voice` | Output voice identifier. Defaults to `"alloy"`. | `"coral"` | [Voice options](https://platform.openai.com/docs/guides/realtime-conversations#voice-options) | -| `params` | OpenAI Realtime session parameters. Audio must remain mono PCM at 24000 Hz. | `{"max_output_tokens": 4096}` | [`session.update`](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update) | -| `connection` | Restart timing overrides. | `{"auto_restart": False}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.ConnectionConfig) | +### Session settings -### Additional Provider Options - -Use direct options such as `voice` and `transcription_model_id` for common settings, and pass additional OpenAI Realtime options through `params`. +Pass OpenAI's [session settings][session-settings] through `params` to customize +the conversation. The adapter uses [server voice activity detection][vad] by default. +This example allows 800 milliseconds of silence before detecting the end of speech: ```python from strands.bidi.models import OpenAIRealtimeModel @@ -95,37 +80,50 @@ from strands.bidi.models import OpenAIRealtimeModel model = OpenAIRealtimeModel( model_id="gpt-realtime-2.1", transcription_model_id="gpt-transcribe", - api_key="", - voice="coral", - params={"audio": {"input": {"turn_detection": {"threshold": 0.3}}}}, + params={"audio": {"input": {"turn_detection": {"silence_duration_ms": 800}}}}, ) ``` -Nested dictionaries in `params` merge with the existing configuration, preserving -unspecified fields. If a setting overlaps with a default or direct option, `params` -takes precedence. - -Calling `update_config(params=...)` replaces the entire `params` dictionary. The new -values take effect on the next `start()` or `restart()`. - -## Connection Restart - -The OpenAI Realtime API exposes no server-side resume handle, so when a connection nears its time limit, `BidiAgent` restarts it and replays the accumulated conversation history into the new connection. The provider then sends a short system instruction so the model continues in the same language without re-introducing itself. The agent emits a [`BidiConnectionRestartEvent`](../events.mdx#connection-restarts) when this happens; treat it as informational rather than an error. - -Set `timeout_s` in the client config to control when the client-side timeout fires ahead of OpenAI's session limit (see the table above). For the full lifecycle, see [Connection restarts](../agent.mdx#connection-restarts). +Use OpenAI's field names inside `params`. Nested settings merge with the defaults, +and `params` takes precedence over direct options such as `voice`. Keep turn +detection, automatic responses, and interruption enabled; the adapter requires them. -## Troubleshooting +## Text output -### Module Not Found +The model produces speech and its transcript by default. To generate text without +audio, set `output_modalities` through `params`: -If you encounter the error `ModuleNotFoundError: No module named 'websockets'`, this means the WebSocket dependency hasn't been properly installed in your environment. To fix this, run `pip install 'strands-agents[bidi-openai]'`. - -### Authentication Errors - -Set the `OPENAI_API_KEY` environment variable or pass the key through `api_key`. +```python +from strands.bidi.models import OpenAIRealtimeModel -## References +model = OpenAIRealtimeModel( + model_id="gpt-realtime-2.1", + transcription_model_id=None, + params={"output_modalities": ["text"]}, +) +``` -- [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime) -- [OpenAI API Reference](https://platform.openai.com/docs/api-reference/realtime) -- [Python API Reference](@api/python/strands.bidi.models#strands.bidi.models.OpenAIRealtimeModel) +For a complete terminal conversation using this configuration, see +[Console I/O](../io.mdx#console-io). + +## Connection restarts + +OpenAI limits Realtime sessions to [60 minutes][session-lifecycle]. The adapter +uses a 50-minute timeout to leave time for reconnection before that limit. By +default, `BidiAgent` schedules a restart after 45 minutes, ahead of this timeout. +Each restart opens a new session and replays conversation history, including text, +completed transcripts, and tool calls and results. + +Set `connection.restart_after_s` to customize the schedule. The `timeout_s` option +controls the adapter's timeout and defaults to its maximum of 3000 seconds (50 minutes). +Keep the scheduled restart earlier than that timeout to allow time for the transition. +See [Connection restarts](../agent.mdx#connection-restarts) for timing and events. + +[realtime]: https://developers.openai.com/api/docs/guides/realtime +[api-keys]: https://platform.openai.com/settings/organization/api-keys +[transcription-model]: https://developers.openai.com/api/docs/models/gpt-transcribe +[voices]: https://developers.openai.com/api/docs/guides/realtime-conversations#voice-options +[session-settings]: https://developers.openai.com/api/reference/resources/realtime/client-events#session.update +[vad]: https://developers.openai.com/api/docs/guides/realtime-vad#server-vad +[session-lifecycle]: https://developers.openai.com/api/docs/guides/realtime-conversations#session-lifecycle-events +[model-api]: @api/python/strands.bidi.models#strands.bidi.models.OpenAIRealtimeModel.__init__