Rewrite voice agent testing best practices

fern/assistants/examples/docs-agent.mdx

@@ -517,121 +517,25 @@ Guidelines:
     </Note>
   </Step>
 
-  <Step title="Create test suite">
-    Create test scenarios to validate your documentation assistant's responses.
-
-    <Tabs>
-      <Tab title="Dashboard">
-        1. Navigate to **Test** > **Voice Test Suites** in your dashboard  
-        2. Click **Create Test Suite**
-        3. Add test scenarios with expected behaviors
-        4. Run tests to validate assistant performance
-      </Tab>
-      <Tab title="TypeScript (Server SDK)">
-        ```typescript
-        import { VapiClient } from "@vapi-ai/server-sdk";
-
-        const vapi = new VapiClient({ token: "YOUR_VAPI_API_KEY" });
-
-        // Create test suite for documentation assistant
-        const testSuite = await vapi.testSuites.create({
-          name: "Documentation Assistant Tests",
-          assistantId: "YOUR_ASSISTANT_ID",
-          testCases: [
-            {
-              name: "Basic greeting test",
-              scenario: "User says hello",
-              expectedBehavior: "Assistant responds with greeting and asks how to help"
-            },
-            {
-              name: "Documentation query test", 
-              scenario: "User asks about API endpoints",
-              expectedBehavior: "Assistant uses docsquery tool and provides relevant information"
-            },
-            {
-              name: "Unknown topic test",
-              scenario: "User asks about unrelated topic",
-              expectedBehavior: "Assistant politely redirects to documentation topics"
-            }
-          ]
-        });
-
-        console.log(`Test suite created with ID: ${testSuite.id}`);
-        console.log("Next: Go to Dashboard to run the test suite");
-        ```
-      </Tab>
-      <Tab title="Python (Server SDK)">
-        ```python
-        import requests
-
-        # Create test suite for documentation assistant
-        url = "https://api.vapi.ai/test-suite"
-        headers = {
-            "Authorization": f"Bearer {YOUR_VAPI_API_KEY}",
-            "Content-Type": "application/json"
-        }
-
-        data = {
-            "name": "Documentation Assistant Tests",
-            "assistantId": "YOUR_ASSISTANT_ID",
-            "testCases": [
-                {
-                    "name": "Basic greeting test",
-                    "scenario": "User says hello", 
-                    "expectedBehavior": "Assistant responds with greeting and asks how to help"
-                },
-                {
-                    "name": "Documentation query test",
-                    "scenario": "User asks about API endpoints",
-                    "expectedBehavior": "Assistant uses docsquery tool and provides relevant information"
-                },
-                {
-                    "name": "Unknown topic test",
-                    "scenario": "User asks about unrelated topic",
-                    "expectedBehavior": "Assistant politely redirects to documentation topics"
-                }
-            ]
-        }
-
-        response = requests.post(url, headers=headers, json=data)
-        test_suite = response.json()
-        print(f"Test suite created with ID: {test_suite['id']}")
-        print("Next: Go to Dashboard to run the test suite")
-        ```
-      </Tab>
-      <Tab title="cURL">
-        ```bash
-        curl -X POST https://api.vapi.ai/test-suite \
-             -H "Authorization: Bearer YOUR_VAPI_API_KEY" \
-             -H "Content-Type: application/json" \
-             -d '{
-               "name": "Documentation Assistant Tests",
-               "assistantId": "YOUR_ASSISTANT_ID", 
-               "testCases": [
-                 {
-                   "name": "Basic greeting test",
-                   "scenario": "User says hello",
-                   "expectedBehavior": "Assistant responds with greeting and asks how to help"
-                 },
-                 {
-                   "name": "Documentation query test", 
-                   "scenario": "User asks about API endpoints",
-                   "expectedBehavior": "Assistant uses docsquery tool and provides relevant information"
-                 },
-                 {
-                   "name": "Unknown topic test",
-                   "scenario": "User asks about unrelated topic", 
-                   "expectedBehavior": "Assistant politely redirects to documentation topics"
-                 }
-               ]
-             }'
-        ```
-      </Tab>
-    </Tabs>
-
-    <Note>
-      Test suites can only be executed through the Dashboard. Navigate to **Test** > **Voice Test Suites** to run your created tests.
-    </Note>
+  <Step title="Test the assistant">
+    Test important decisions and complete conversation outcomes before sharing
+    the assistant with users.
+
+    <CardGroup cols={2}>
+      <Card title="Test decisions with Evals" icon="clipboard-check" href="/test/evals-best-practices">
+        Check whether the assistant uses the documentation tool, answers from
+        the result, and redirects requests outside its scope.
+      </Card>
+      <Card title="Test outcomes with Simulations" icon="flask-vial" href="/test/simulations-best-practices">
+        Run complete conversations with different questions and caller
+        behaviors, then check whether the user gets a useful answer.
+      </Card>
+    </CardGroup>
+
+    Start with a greeting, a supported documentation question, an ambiguous
+    question, a tool error, and an unrelated request. See
+    [testing voice agents](/test/voice-testing) for guidance on building and
+    maintaining coverage.
   </Step>
 
   <Step title="Create a web component">

fern/calls/call-ended-reason.mdx

@@ -218,5 +218,5 @@ For SIP trunk setup issues, see [Troubleshoot SIP trunk credential errors](/adva
 ## Next steps
 
 - **[Troubleshoot call errors](/calls/troubleshoot-call-errors):** Step-by-step diagnosis guide organized by what the caller experienced.
-- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, and test suites.
+- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, Simulations, and Evals.
 - **[How to report issues](/issue-reporting):** Include your `call_id` and account email when contacting support.

fern/calls/troubleshoot-call-errors.mdx

@@ -266,7 +266,7 @@ These are not errors — they indicate the call ended as expected.
 ## Next steps
 
 - **[Call end reasons](/calls/call-ended-reason):** Complete reference of every `endedReason` code.
-- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, and test suites.
+- **[Debugging voice agents](/debugging):** General debugging workflow using dashboard tools, logs, Simulations, and Evals.
 - **[Debug forwarding drops](/phone-calling/in-call-control/transfer-calls/debug-forwarding-drops):** Deep dive into transfer failures.
 - **[Troubleshoot SIP trunk errors](/advanced/sip/troubleshoot-sip-trunk-credential-errors):** Resolve SIP credential validation failures.
 - **[Troubleshoot SIP response codes](/advanced/sip/troubleshoot-sip-response-codes):** Identify where a SIP request failed and what to check next.

fern/changelog/2025-02-25.mdx

@@ -1,6 +1,6 @@
 ## Test Suite APIs, Enhanced Call Transfers, Voice Model Enhancements
 
-1.  **Introducing Test Suite Management APIs:** You can now test your assistant conversations before deploying them by creating [end-to-end tests](https://docs.vapi.ai/test/voice-testing#step-1-create-a-new-test-suite), [adding test cases](https://docs.vapi.ai/test/voice-testing#step-3-add-test-cases), and [running and reviewing test suites](https://docs.vapi.ai/test/voice-testing#step-5-run-and-review-tests). You can configure these tests through the [Test Suites dashboard page](https://dashboard.vapi.ai/test-suites) and [Test Suite APIs](https://docs.vapi.ai/api-reference/test-suites/test-suite-controller-find-all-paginated), and learn more in the [docs](https://docs.vapi.ai/test/voice-testing).
+1.  **Introducing Test Suite Management APIs:** You can now create, run, and review end-to-end assistant tests through the Test Suites dashboard and APIs. Test Suites are now deprecated; use [Simulations](/observability/simulations-overview) for current testing guidance.
 
 <Frame caption="Test Suite Management APIs">
     <img src="../static/images/changelog/test-suite-management.png" alt="Test Suite Management APIs" />

fern/debugging.mdx

@@ -58,11 +58,14 @@ Start with these immediate checks before diving deeper:
   <Step title="Test individual components">
     Use [dashboard](https://dashboard.vapi.ai/) testing features:
 
-    <CardGroup cols={2}>
-      <Card title="Voice Test Suites" icon="vial">
-        Automated testing for assistants
+    <CardGroup cols={3}>
+      <Card title="Simulations" icon="flask-vial" href="/observability/simulations-overview">
+        Test outcomes across complete conversations
+      </Card>
+      <Card title="Evals" icon="clipboard-check" href="/observability/evals-quickstart">
+        Test decisions at specific conversation points
       </Card>
-      <Card title="Tool Testing" icon="wrench">
+      <Card title="Tool Testing" icon="wrench" href="#tool-testing">
         Test tools with sample data
       </Card>
     </CardGroup>
@@ -132,15 +135,14 @@ vapi listen --forward-to localhost:3000/webhook
 `vapi listen` is a local forwarder that requires a separate tunneling service. Update your webhook URLs in Vapi to use the tunnel's public URL. [Learn more →](/cli/webhook)
 </Note>
 
-### Voice Test Suites
+### Simulations and Evals
 
-Navigate to `Test > Voice Test Suites` to:
-- Run automated tests on your assistants
-- Test conversation flows with predefined scenarios
-- Verify assistant behavior across different inputs
-- Monitor performance over time
+Use **Simulations** to reproduce a problem across a complete conversation and
+check the final outcome. Use **Evals** to isolate a specific decision, such as
+choosing a tool, asking for missing information, or escalating the call.
 
-<video autoPlay loop muted src="./static/videos/debugging/voice-test-suites.mp4" type="video/mp4" style={{ aspectRatio: '16 / 9', width: '100%' }} />
+See [testing voice agents](/test/voice-testing) for guidance on choosing a test
+and turning production failures into regression coverage.
 
 ### Tool Testing
 

fern/docs-agent-prompt.txt

@@ -54,7 +54,7 @@ Vahpi: Key Context & API Overviews High-Level
 * Assistants: POST /assistant to create or configure an LLM-based voice agent.
 * Tools: POST /tool to define calls to external APIs or built-in call-management (like transferCall, endCall, sms, etc.).
 * Workflows: [BETA] multi-node branching logic.
-* Test Suites: run voice or chat-based test scripts for agent QA. When referencing endpoints or doc sections, do so lightly, e.g.:
+* Testing: use Simulations for complete conversations and outcomes. Use Evals for decisions at specific points in a mock conversation. Do not recommend the deprecated Test Suites product. When referencing endpoints or doc sections, do so lightly, e.g.:
 * "You can see the Create Call (POST /call) docs to set up an outbound call."
 * "To add a custom transcriber, look at the 'custom transcriber' section in the docs."
 
@@ -97,13 +97,11 @@ Parameter & Configuration Structure Below is a thorough breakdown of possible pa
         * nodes: array of conversation nodes, each can hold a sub-config for model, transcriber, voice.
         * edges: specify which node leads to which, with optional conditions or metadata.
         * model: default LLM for the entire workflow if not overridden at the node level.
-6. Test Suites / Test Creation
-    * Purpose: Automated QA for your voice or chat assistants.
-    * Key Fields:
-        * type: "voice" or "chat".
-        * script: scenario or conversation steps.
-        * scorers: define how success is measured (like an AI-based rubric).
-        * numAttempts: how many times it can re-run. (Additional endpoints exist, but these are the most common. Use docs_query if you need specifics.)
+6. Simulations and Evals
+    * Simulations test outcomes across complete conversations with an AI tester. They can run in chat or voice mode and use success criteria to grade the result.
+    * Evals test decisions at specific points in a mock conversation. Use them for response, tool-call, and tool-argument checks.
+    * Start with Evals and chat Simulations while iterating. Use voice Simulations for representative end-to-end checks before release.
+    * Link to the testing best-practices and Observability guides for current setup details. Never recommend the deprecated Test Suites product or `/test-suite` endpoints.
 I
 nteraction Flow When a user asks:
 1. Identify if it's a doc-type question (like "How do I create an outbound call?") → Use docs_helper mode.
@@ -135,4 +133,4 @@ Final Requirements
 * No raw JSON or code.
 * If uncertain, politely note that you're missing info, suggest potential next steps or a doc reference.
 
-Additional Handling for "How do I get started?" If the user says something like "How do I get started?", ask if they're a developer or not. If they're a technical user, invite them to look at the API reference (assistants, calls, tools) or the Web SDK for adding an agent into their site or app. If they're not a developer, guide them to the Dashboard Quickstart, explaining how they can create and configure a voice agent step by step without coding. 
+Additional Handling for "How do I get started?" If the user says something like "How do I get started?", ask if they're a developer or not. If they're a technical user, invite them to look at the API reference (assistants, calls, tools) or the Web SDK for adding an agent into their site or app. If they're not a developer, guide them to the Dashboard Quickstart, explaining how they can create and configure a voice agent step by step without coding.

fern/docs.yml

@@ -647,15 +647,21 @@ navigation:
             collapsed: true
             icon: fa-light fa-clipboard-check
             contents:
-              - page: Test suites
-                path: test/test-suites.mdx
-                icon: fa-light fa-check
-              - page: Chat testing
-                path: test/chat-testing.mdx
-                icon: fa-light fa-message
-              - page: Voice testing
+              - page: Testing voice agents
                 path: test/voice-testing.mdx
                 icon: fa-light fa-volume-high
+              - page: Plan test coverage
+                path: test/plan-test-coverage.mdx
+                icon: fa-light fa-list-check
+              - page: Test outcomes with Simulations
+                path: test/simulations-best-practices.mdx
+                icon: fa-light fa-flask-vial
+              - page: Test decisions with Evals
+                path: test/evals-best-practices.mdx
+                icon: fa-light fa-clipboard-check
+              - page: Run and maintain tests
+                path: test/run-and-maintain-tests.mdx
+                icon: fa-light fa-arrows-rotate
 
       - section: Resources
         collapsed: open-by-default
@@ -1013,6 +1019,10 @@ navigation:
   - tab: changelog
 
 redirects:
+  - source: /test/test-suites
+    destination: /observability/simulations-quickstart
+  - source: /test/chat-testing
+    destination: "/test/simulations-best-practices#choose-chat-or-voice-deliberately"
   - source: /tools/code-tool
     destination: /tools/api-request
   - source: /security-and-privacy/retrieve-call-artifacts

fern/observability/evals-advanced.mdx

@@ -6,7 +6,10 @@ slug: observability/evals-advanced
 
 ## Overview
 
-This guide covers advanced evaluation strategies, testing patterns, and best practices for building robust test suites that ensure your AI agents work reliably in production.
+This guide covers advanced evaluation strategies, testing patterns, and best practices for building robust test coverage that helps your AI agents work reliably in production.
+
+For operator-focused guidance on choosing checkpoints and writing durable
+checks, see [test decisions with Evals](/test/evals-best-practices).
 
 **You'll learn:**
 
@@ -54,7 +57,7 @@ Quick validation that core functionality works. Run these first to catch obvious
 
 **When to use:**
 
-- Before running expensive test suites
+- Before running broader test sets
 - After deploying configuration changes
 - As health checks in monitoring
 - Quick validation during development
@@ -677,7 +680,7 @@ Is the exact wording critical?
 Stops test immediately when critical validation fails.
 
 2. **Run critical tests first:**
-   Organize test suites so smoke tests and critical validations run before expensive tests.
+   Organize evaluations so smoke tests and critical validations run before expensive tests.
 
 3. **Keep conversations focused:**
    Aim for 5-10 turns maximum. Split longer scenarios into multiple tests.
@@ -729,7 +732,7 @@ Store evaluation definitions alongside your codebase:
 {" "}
 
 <Step title="Monthly: Audit test coverage">
-  Review test suite completeness: - All critical user flows covered? - New
+  Review test coverage: - All critical user flows covered? - New
   features have tests? - Deprecated features removed?
 </Step>
 

fern/observability/evals-quickstart.mdx

@@ -8,6 +8,9 @@ slug: observability/evals-quickstart
 
 This quickstart guide will help you set up automated testing for your AI assistants and squads. In just a few minutes, you'll create mock conversations, define expected behaviors, and validate your agents work correctly before production.
 
+For operator-focused guidance on choosing checkpoints and writing durable
+checks, see [test decisions with Evals](/test/evals-best-practices).
+
 <div style="position: relative; padding-bottom: 56.25%; height: 0;"><iframe style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border: 0;" src="https://www.tella.tv/video/cmgu6muyb002m0bktda8r6nou/embed?b=0&title=0&a=1&loop=0&t=0&muted=0&wt=1" allowfullscreen allowtransparency></iframe></div>
 
 ### What are Evals?
@@ -1306,7 +1309,7 @@ Validate smooth transitions between squad members:
 }
 ```
 
-### Regression test suite
+### Regression test set
 
 Organize related tests for systematic validation:
 

fern/observability/monitoring-quickstart.mdx

@@ -699,7 +699,23 @@ curl -X PATCH "https://api.vapi.ai/monitoring/issue/e5f6a7b8-c9d0-1234-efab-5678
     icon="flask"
     href="/observability/evals-quickstart"
   >
-    Test your assistants before deployment with automated evaluations
+    Test decisions at specific conversation points
+  </Card>
+
+  <Card
+    title="Simulations"
+    icon="flask-vial"
+    href="/observability/simulations-overview"
+  >
+    Test outcomes across complete conversations
+  </Card>
+
+  <Card
+    title="Testing voice agents"
+    icon="clipboard-check"
+    href="/test/voice-testing"
+  >
+    Plan testing coverage before and after launch
   </Card>
 </CardGroup>