Repository navigation
[Issue] azure.yaml examples in agent docs aren't validated and can ship broken #9330
Description
- Make sure you've installed the latest version using instructions
Output from azd version
N/A — this is about docs correctness in the repo, reproduced against azure.ai.agents extension 1.0.0-beta.7.
Describe the bug
azure.yaml examples in extension docs aren't validated by anything, so they can drift out of sync with the code and ship in a state where copying them fails. Two separate instances of this were found recently, both the same class of defect:
cli/azd/extensions/azure.ai.agents/README.md— the "Migrating Legacy Agent Configuration"After:example omitted the required agentname. Copying it fails withname cannot be empty(fixed in fix: name the azure.yaml key in the RAI policy validation error #9328).- The published Learn article Add a content safety guardrail to a hosted agent documented
rai_config.rai_policy_nameon theazure.ai.agentservice, whichazddoesn't support at all. Becauseazdignores unrecognized service properties, the agent deployed with no guardrail and no error (fix: MicrosoftDocs/azure-ai-docs-pr#13567).
Both were found by hand. Nothing in CI would have caught either.
To Reproduce
Run any doc azure.yaml agent service snippet through the resolver:
props, _ := structpb.NewStruct(map[string]any{
"kind": "hosted",
// no "name"
})
_, _, _, _, err := AgentDefinitionFromService(&azdext.ServiceConfig{
Name: "my-agent", Host: "azure.ai.agent", AdditionalProperties: props,
})
// err = agent service definition is not valid: validation failed:
// - template.name not in valid format: name cannot be emptyNote the service key (my-agent) is not substituted into the definition's name.
Expected behavior
A doc example that a user copies into azure.yaml should be valid, and drift should be caught automatically rather than by review.
Additional context
Two approaches were discussed while fixing #9328, neither taken there to keep that PR scoped:
-
Validate doc snippets in a test. Extract fenced
```yamlblocks containinghost: azure.ai.agentfromREADME.md/docs/*.mdand run each throughAgentDefinitionFromService. Catches real regressions, but couples tests to prose formatting and would need an opt-out marker for deliberately partial snippets (see below). -
Make
schemas/examples/*.azure.yamlthe single source of truth. Those files are already complete and correct (simple.azure.yamlcorrectly setsname: assistant). Docs would link to or embed from them instead of hand-writing YAML. Less fragile, but a larger docs restructure.
One caveat for either approach: not every doc snippet is meant to be complete. The three azure.ai.agent entries in cli/azd/extensions/azure.ai.agents/docs/private-networking.md intentionally omit kind, so AgentDefinitionFromService returns found=false, err=nil and azd falls back to the on-disk definition. Those are fine as-is and a naive validator would produce false positives on them.
Related: #9328, MicrosoftDocs/azure-ai-docs-pr#13567
Activity
test(azure.ai.agents): validate azure.yaml examples in docs (Azure#9368)