Skip to content

[Issue] azure.yaml examples in agent docs aren't validated and can ship broken #9330

Closed

Description

@huimiu
  • Make sure you've installed the latest version using instructions

Output from azd version

N/A — this is about docs correctness in the repo, reproduced against azure.ai.agents extension 1.0.0-beta.7.

Describe the bug

azure.yaml examples in extension docs aren't validated by anything, so they can drift out of sync with the code and ship in a state where copying them fails. Two separate instances of this were found recently, both the same class of defect:

  1. cli/azd/extensions/azure.ai.agents/README.md — the "Migrating Legacy Agent Configuration" After: example omitted the required agent name. Copying it fails with name cannot be empty (fixed in fix: name the azure.yaml key in the RAI policy validation error #9328).
  2. The published Learn article Add a content safety guardrail to a hosted agent documented rai_config.rai_policy_name on the azure.ai.agent service, which azd doesn't support at all. Because azd ignores unrecognized service properties, the agent deployed with no guardrail and no error (fix: MicrosoftDocs/azure-ai-docs-pr#13567).

Both were found by hand. Nothing in CI would have caught either.

To Reproduce

Run any doc azure.yaml agent service snippet through the resolver:

props, _ := structpb.NewStruct(map[string]any{
    "kind": "hosted",
    // no "name"
})
_, _, _, _, err := AgentDefinitionFromService(&azdext.ServiceConfig{
    Name: "my-agent", Host: "azure.ai.agent", AdditionalProperties: props,
})
// err = agent service definition is not valid: validation failed:
//         - template.name not in valid format: name cannot be empty

Note the service key (my-agent) is not substituted into the definition's name.

Expected behavior

A doc example that a user copies into azure.yaml should be valid, and drift should be caught automatically rather than by review.

Additional context

Two approaches were discussed while fixing #9328, neither taken there to keep that PR scoped:

  1. Validate doc snippets in a test. Extract fenced ```yaml blocks containing host: azure.ai.agent from README.md / docs/*.md and run each through AgentDefinitionFromService. Catches real regressions, but couples tests to prose formatting and would need an opt-out marker for deliberately partial snippets (see below).

  2. Make schemas/examples/*.azure.yaml the single source of truth. Those files are already complete and correct (simple.azure.yaml correctly sets name: assistant). Docs would link to or embed from them instead of hand-writing YAML. Less fragile, but a larger docs restructure.

One caveat for either approach: not every doc snippet is meant to be complete. The three azure.ai.agent entries in cli/azd/extensions/azure.ai.agents/docs/private-networking.md intentionally omit kind, so AgentDefinitionFromService returns found=false, err=nil and azd falls back to the on-disk definition. Those are fine as-is and a naive validator would produce false positives on them.

Related: #9328, MicrosoftDocs/azure-ai-docs-pr#13567

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area/docsDocumentation, design docsarea/extensionsExtensions (general)bugSomething isn't workingext-agentsazure.ai.agents extension

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions