Skip to content

docs(bidi): refine model provider guides [replica of harness-sdk#4894] #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
poshinchen wants to merge 1 commit into
base: base-pr-4894
Choose a base branch
from
Open

docs(bidi): refine model provider guides [replica of harness-sdk#4894] #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 77 additions & 121 deletions site/src/content/docs/user-guide/sdk/bidi/models/bedrock.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,168 +1,124 @@
---
title: Bedrock Nova Sonic
description: 'Build real-time speech-to-speech agents with Amazon Bedrock Nova Sonic and Strands. Bidirectional audio streaming with barge-in handling and tool calling.'
description: "Configure Amazon Nova Sonic for BidiAgent: set AWS credentials, choose a voice, tune turn detection, and preserve context across connection restarts."
languages: [python]
tags: [bidi-streaming, aws, bedrock]
sourceLinks:
- path: strands-py/src/strands/bidi/models/bedrock.py
- path: strands-py/src/strands/bidi/models/configs.py
redirectFrom:
- docs/user-guide/concepts/bidirectional-streaming/models/nova_sonic
- docs/user-guide/sdk/bidirectional-streaming/models/bedrock
---

[Amazon Nova Sonic][nova-sonic] brings expressive, multilingual voice conversations
to your agents. Users can speak naturally, interrupt a reply, and ask the agent to
take action through tools.

[Amazon Nova Sonic](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html) provides real-time, conversational interactions through bidirectional audio streaming. Amazon Nova Sonic processes and responds to real-time speech as it occurs, enabling natural, human-like conversational experiences. Key capabilities and features include:

- Adaptive speech response that dynamically adjusts delivery based on the prosody of the input speech.
- Graceful handling of user barge-ins without dropping conversational context.
- Function calling and agentic workflow support for building complex AI applications.
- Robustness to background noise for real-world deployment scenarios.
- Multilingual support with expressive voices and speaking styles. Expressive voices are offered, including both masculine-sounding and feminine sounding, in seven languages: English (US, UK, AU, IN), French, Italian, German, Spanish (US), Portuguese (BR), and Hindi.
- Recognition of varied speaking styles across all supported languages.
Follow the [quickstart](../quickstart.mdx) to build a voice agent that listens,
speaks, and calls tools.

## Installation

:::caution[Python 3.12+ Required]
Nova Sonic requires Python 3.12 or higher due to its experimental AWS SDK dependency.
:::note[Python version]
Nova Sonic requires Python 3.12 or later.
:::

Nova Sonic is included in the base bidirectional streaming dependencies for Strands Agents.

To install it, run:
Install the Nova Sonic extra:

```bash
pip install 'strands-agents[bidi,bidi-io,bidi-pyaudio]'
pip install "strands-agents[bidi]"
```

Or to install all bidirectional streaming providers at once:
For local audio dependencies and device setup, see
[Audio I/O](../io.mdx#audio-io).

```bash
pip install 'strands-agents[bidi-all,bidi-pyaudio]'
```
## Credentials

## Usage
Configure AWS credentials with permission to invoke Nova Sonic in a
[supported region][nova-regions].
`BedrockNovaSonicModel` uses Boto3's [credential chain][boto3-credentials], including
environment variables, named profiles, and IAM roles. See
[Setting up AWS credentials][credentials-setup]
for shared setup instructions.

After installing the Bedrock Nova Sonic and local audio extras, create a voice agent:
To use a named profile, pass a Boto3 session with its region:

```python
import asyncio
import boto3

from strands.bidi.agent import BidiAgent
from strands.bidi.io import AudioIO
from strands.bidi.models import BedrockNovaSonicModel
from strands.vended_tools import notebook


async def main() -> None:
model = BedrockNovaSonicModel(
model_id="amazon.nova-2-sonic-v1:0",
region="us-east-1",
voice="tiffany",
)
agent = BidiAgent(model=model, tools=[notebook])
audio_io = AudioIO()
await agent.run(inputs=[audio_io.input()], outputs=[audio_io.output()])


if __name__ == "__main__":
asyncio.run(main())
```

## Cross-Modal Input

Nova Sonic accepts text input at any point during an active voice session, without interrupting or waiting on the audio stream. Send [text input](../content.mdx#send-input) the same way you would outside of a live conversation:

```python
await agent.send({"text": "What's the weather in Seattle?"})
model = BedrockNovaSonicModel(
model_id="amazon.nova-2-sonic-v1:0",
boto_session=boto3.Session(profile_name="voice-agent", region_name="us-east-1"),
)
```

Text and audio input can be interleaved freely. Sending text does not require pausing
the microphone or waiting for the model to finish speaking.

## Credentials
Without a custom session, you can pass `region` directly. If omitted, the model
uses Boto3's configured region, falling back to `us-east-1`.

:::caution[Nova Sonic is available in us-east-1, us-west-2, eu-north-1, and ap-northeast-1.]
:::

Nova Sonic requires AWS credentials for access. `BedrockNovaSonicModel` uses an experimental [Bedrock client](https://github.com/aws/aws-sdk-python/tree/develop/clients/aws-sdk-bedrock-runtime/src/aws_sdk_bedrock_runtime), which accepts credentials in the following ways:

**Option 1: Environment Variables**

```bash
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_SESSION_TOKEN=your_session_token # If using temporary credentials
export AWS_REGION=your_region_name
```
## Configuration

**Option 2: Boto3 Session**
Create a model with the voice and audio settings for your application, then pass it
to `BidiAgent`. This example selects the `tiffany` voice and 24 kHz output audio:

```python
import boto3
from strands.bidi.agent import BidiAgent
from strands.bidi.models import BedrockNovaSonicModel


boto_session = boto3.Session(
aws_access_key_id="your_access_key",
aws_secret_access_key="your_secret_key",
aws_session_token="your_session_token", # If using temporary credentials
region_name="your_region_name",
profile_name="your_profile" # Optional: Use a specific profile
)
model = BedrockNovaSonicModel(
model_id="amazon.nova-2-sonic-v1:0",
boto_session=boto_session,
region="us-east-1",
voice="tiffany",
audio={"output": {"sample_rate": 24000}},
)
agent = BidiAgent(model=model)
```

For more details on this approach, please refer to the [boto3 session docs](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html).

## Configuration

### Client Options

| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `boto_session` | A `boto3.Session` instance under which AWS credentials are configured. | `None` |
| `region` | Region under which credentials are configured. Cannot use if providing `boto_session`. | `us-east-1` |

### Model Config

| Parameter | Description | Example | Options |
| --------- | ----------- | ------- | ------- |
| `model_id` | Nova Sonic model identifier. | `"amazon.nova-2-sonic-v1:0"` | Nova Sonic model IDs |
| `audio` | Input and output stream options. | `{"output": {"sample_rate": 24000}}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicAudioConfig) |
| `voice` | Output voice identifier. Defaults to `"matthew"`. | `"tiffany"` | Nova Sonic voices |
| `params` | Provider-specific session parameters, such as inference and turn detection configuration. | `{"inferenceConfiguration": {"temperature": 0.7}}` | [`sessionStart` fields](https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-input-events.html) |
| `connection` | Restart timing overrides. | `{"auto_restart": False}` | [reference](@api/python/strands.bidi.models#strands.bidi.models.ConnectionConfig) |

:::caution[Conversation History Limits]
Nova Sonic caps conversation history passed via `messages` at 50KB per message and 200KB total. Messages exceeding the per-message limit are truncated; if the total exceeds 200KB, the oldest messages are dropped until it fits. This happens silently (only visible via debug-level logs) and only applies to history provided at connection start, not to turns generated during the live conversation.
:::

## Connection Restart

Nova Sonic keeps a single connection open for about 8 minutes. Before the connection reaches that limit, `BidiAgent` restarts it and replays the conversation history into the new connection, so the conversation continues without a break. The agent emits a [`BidiConnectionRestartEvent`](../events.mdx#connection-restarts) when this happens; treat it as informational rather than an error.

See [Connection restarts](../agent.mdx#connection-restarts) for the full lifecycle.

## Troubleshooting
Choose a voice from Nova Sonic's [voices and languages][nova-voices]. Audio uses
mono PCM, with 16 kHz input and output by default. Set either stream's sample rate
through [`audio`][audio-config].

### Module Not Found

If you encounter the error `ModuleNotFoundError: No module named 'aws_sdk_bedrock_runtime'`, this means the experimental Bedrock runtime dependency hasn't been properly installed in your environment. To fix this, run `pip install 'strands-agents[bidi]'`.

:::note[Python Version Requirement]
Nova Sonic requires Python 3.12+ due to the experimental AWS SDK dependency. If you're using an older Python version, you'll need to upgrade.
:::
See the [`BedrockNovaSonicModel` API reference][model-api] for all constructor
options.

### Hanging
### Session settings

When credentials are misconfigured, the model provider does not throw an exception (a quirk of the underlying experimental Bedrock client). As a result, the provider allows the user to proceed forward with a call to `receive`, which emits no events and thus presents an indefinite hanging behavior.
Pass Nova Sonic's [session settings][nova-input-events] through `params` to tune
inference and turn detection. For example, `LOW` endpointing sensitivity allows
longer pauses before the model decides the user has finished speaking:

As a reminder, Nova Sonic is only available in us-east-1, us-west-2, eu-north-1, and ap-northeast-1.
```python
from strands.bidi.models import BedrockNovaSonicModel

## References
model = BedrockNovaSonicModel(
model_id="amazon.nova-2-sonic-v1:0",
params={"turnDetectionConfiguration": {"endpointingSensitivity": "LOW"}},
)
```

- [Nova Sonic](https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html)
- [Experimental Bedrock Client](https://github.com/aws/aws-sdk-python/tree/develop/clients/aws-sdk-bedrock-runtime)
- [Python API Reference](@api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicModel)
Use the provider's field names inside `params`, as shown above.

## Connection restarts

Nova Sonic limits each connection to eight minutes. By default, `BidiAgent`
schedules a restart after seven minutes and replays conversation history into the
new connection. See [Connection restarts](../agent.mdx#connection-restarts) to
customize the timing and track the transition.

On startup and restart, the adapter replays text from the
[conversation history][nova-history],
including [completed transcripts](../events.mdx#messages). It truncates individual
messages to 50 KiB and retains up to 200 KiB of recent history. These limits apply
to the history sent when opening a connection.

[nova-sonic]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/using-conversational-speech.html
[nova-regions]: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-amazon-nova-2-sonic.html
[nova-voices]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-language-support.html
[nova-input-events]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-input-events.html
[nova-history]: https://docs.aws.amazon.com/nova/latest/nova2-userguide/sonic-chat-history.html
[boto3-credentials]: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
[credentials-setup]: ../../model-providers/amazon-bedrock.mdx#setting-up-aws-credentials
[audio-config]: @api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicAudioConfig
[model-api]: @api/python/strands.bidi.models#strands.bidi.models.BedrockNovaSonicModel.__init__
Loading
Loading