feat(agents): structured output - #574
Draft
MarioCadenas wants to merge 8 commits into
Draft
Conversation
Add an optional `output?: z.ZodType` field to AgentDefinition and make
createAgent generic so `createAgent({ output: Schema })` returns
`AgentDefinition<z.infer<Schema>>`. Thread the type param through
RunAgentResult (new `output?: TOutput`) and store the runtime schema on
RegisteredAgent. No behavior yet — this is the typing surface only.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add an optional `outputSchema` (JSON Schema) to AgentInput. When set and
the completion is tool-free, DatabricksAdapter sends it as an
OpenAI-compatible `response_format: { type: json_schema, strict: true }`,
so the final text is the JSON to validate. Tool-having runs are unchanged
(Claude rejects response_format + tools) and get re-formatted by a separate
tool-free structuring pass — a second run() with no tools.
On a 400 that names the param, strip response_format and retry once (the
Zod validation upstream is the real guarantee). In structured mode the
text-based tool-call fallback is skipped so array-typed JSON isn't misread
as a tool call.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Add resolveStructuredOutput (core/agent): parse + zod-validate the agent's
answer, re-prompting a tool-free structuring pass with the flattened zod
issues up to 2 times before throwing StructuredOutputError with the last
raw output. Tool-free answers are validated inline; tool-having answers get
a first structuring pass. Strips a stray Markdown code fence before parsing.
Wire the in-process runAgent to it via a new 3rd { output } options arg
(overrides the agent's own schema), reusing the adapter for the tool-free
structuring pass. StructuredOutputError joins the error taxonomy; lastRaw is
server-side only and never leaks via clientMessage.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
…event Store the agent's output schema on RegisteredAgent and, when set, resolve structured output after the run's final text: /invocations and /responses gain a top-level output_parsed field (spread like mlflow_trace_id), and /chat emits one final structured_output AgentEvent -> appkit.structured_output SSE event after the streamed text. Retries stay invisible to the stream. The structuring pass runs as a fresh tool-free adapter.run() constrained by the schema, wrapped in a TOOL span nested under the turn's AGENT span. Adds the new event to the shared AgentEvent/ResponseStreamEvent unions and the event translator. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Thread the agent's JSON schema into the main adapter.run() (runAgent + both plugin surfaces) so a tool-free structured agent gets response_format on its own completion — the answer is JSON directly, no wasted structuring round-trip. The adapter ignores outputSchema when tools are present, so tool-having agents are unaffected and still use the separate structuring pass. Completes the tool-branch strategy from the earlier adapter commit. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Split runAgent into two overloads: the no-override call types the result
from the agent's own schema; the per-call { output } override drives the
result type via z.infer of that schema (it also takes precedence at runtime).
Without this, an agent with no schema is AgentDefinition<string>, so a
{ output } override couldn't retype RunAgentResult.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Unit tests for the resolver (validate/retry/exhaust/fence-strip/no-leak), the adapter (response_format on tool-free runs, none with tools, 400 strip-retry), runAgent end-to-end (tool-free inline, tool-having structuring pass, per-call override, throw), and the structured_output event translation. Add a 'Structured output' section to the agents plugin doc and a tool-free 'classifier' example agent to dev-playground. Not verified against a live serving endpoint — needs manual dogfood. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
Contributor
📦 Bundle size reportCompared against
|
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 1.1 MB (+12 KB) | 386 KB (+3.9 KB) |
| Type declarations | 392 KB (+6.4 KB) | 139 KB (+2.6 KB) |
| Source maps | 2.2 MB (+29 KB) | 724 KB (+9.3 KB) |
| Other | 11 KB | 3.7 KB |
| Total | 3.6 MB (+48 KB) | 1.2 MB (+16 KB) |
Per-entry composition (own code — deps external (as shipped))
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
. |
95 KB (+107 B) | 2.5 KB | 98 KB (+107 B) | external | 313 KB (+239 B) |
./beta |
78 KB (+1.4 KB) | 456 B | 78 KB (+1.4 KB) | external | 235 KB (+4.8 KB) |
./testing |
17 KB (-2 B) | 0 B | 17 KB (-2 B) | external | 51 KB |
./tsdown |
520 B | 0 B | 520 B | external | 813 B |
./type-generator |
23 KB | 0 B | 23 KB | external | 65 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
. |
index.js |
initial | 91 KB |
. |
utils.js |
initial | 4.0 KB |
. |
remote-tunnel-manager.js |
lazy | 2.5 KB |
./beta |
beta.js |
initial | 61 KB |
./beta |
stream-manager.js |
initial | 5.8 KB |
./beta |
databricks.js |
initial | 3.6 KB |
./beta |
wide-event-emitter.js |
initial | 3.2 KB |
./beta |
configuration.js |
initial | 2.1 KB |
./beta |
service-context.js |
initial | 1.3 KB |
./beta |
client.js |
initial | 434 B |
./beta |
client-options.js |
initial | 219 B |
./beta |
supervisor-api.js |
lazy | 193 B |
./beta |
databricks.js |
lazy | 141 B |
./beta |
index.js |
lazy | 122 B |
./testing |
index.js |
initial | 17 KB |
./tsdown |
index.js |
initial | 520 B |
./type-generator |
index.js |
initial | 23 KB |
@databricks/appkit-ui
npm tarball (packed): 350 KB — gzipped download (dist + bin; excludes release-only docs/NOTICE).
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 395 KB | 132 KB |
| Type declarations | 229 KB | 84 KB |
| Source maps | 766 KB | 253 KB |
| CSS | 16 KB | 3.2 KB |
| Total | 1.4 MB | 473 KB |
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
./js |
5.3 KB | 49 KB | 55 KB | 208 KB | 14 KB |
./js/beta |
20 B | 0 B | 20 B | 0 B | 0 B |
./react |
432 KB | 49 KB | 481 KB | 1.3 MB | 177 KB |
./react/beta |
1.0 KB | 0 B | 1.0 KB | 0 B | 1.9 KB |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
./js |
index.js |
initial | 5.2 KB |
./js |
chunk |
initial | 120 B |
./js |
apache-arrow |
lazy | 49 KB |
./js/beta |
beta.js |
initial | 20 B |
./react |
index.js |
initial | 430 KB |
./react |
tslib |
initial | 2.1 KB |
./react |
apache-arrow |
lazy | 49 KB |
./react/beta |
beta.js |
initial | 1.0 KB |
Contributor
🤖 AppKit PR bot🔬 Run evalsStart an eval for this PR from the evals-monitor app: Go to Evals Monitor → 📦 Try this PR's app templateScaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh run download 33779428678 -R databricks/appkit -n appkit-template-0.71.0-pr.9feaff1-feat-agent-structured-output-574 -D appkit-pr-574 \
&& unzip -o "appkit-pr-574/appkit-template-0.71.0-pr.9feaff1-feat-agent-structured-output-574.zip" -d "appkit-pr-574" \
&& databricks apps init --template "appkit-pr-574"The template pins |
Databricks rejects response_format under stream:true ("Structured output is
not currently supported with streaming"), and DatabricksAdapter is
streaming-only, so the original inline/streaming approach failed on every
call. Structured output now runs as a dedicated NON-streaming completion:
- Add a non-streaming queryBody transport (connectors/serving query()) and a
structuredCompletion() path in the adapter; run() uses it when outputSchema
is set and tools are empty, emitting the JSON as one message event.
- The main agent run always streams normally (revert the toolless-inline
wiring), so /chat still streams the visible answer; structured output is a
separate pass afterwards.
- resolveStructuredOutput validates the answer as-is first (cheap pre-check),
else reformats via the pass; drop the hadTools branch.
- Broaden the response_format-rejection detector to catch INVALID_PARAMETER_VALUE
/ "structured output" phrasing so strip-and-retry actually fires.
Found via live dogfood; updates tests, docs, and the example accordingly.
Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structured output for AppKit agents
Lets a code-config agent return a typed object validated against a Zod schema instead of freeform text.
createAgent({ output: Schema })(or a per-callrunAgent(agent, input, { output })override) makes the final answer schema-validated and surfaces it on every path.What it does
AgentDefinition<TOutput>+output?: z.ZodType;createAgent<S>is generic so the in-process result is typed viaz.infer. Reuses the existingtoToolJSONSchema; no new converter or dependency./chat), then structured output is produced by a dedicated non-streaming completion constrained by the schema viaresponse_format. The answer is validated as-is first (cheap pre-check); otherwise the pass reformats it.resolveStructuredOutputparses + Zod-validates, re-prompting with the flattened issues up to 2 times, then throwsStructuredOutputError(carryinglastRaw, server-side only). Never returns partial/unvalidated data. If an endpoint rejectsresponse_format, it's stripped and retried (prompt + Zod).output_parsedon the/invocations+/responsesenvelope; a finalappkit.structured_outputSSE event on/chat(after the streamed text; retries invisible);RunAgentResult.outputin-process. The structuring pass is a TOOL span under the turn's AGENT span.classifierexample agent in dev-playground.A live dogfood against a Databricks Claude endpoint returned
INVALID_PARAMETER_VALUE: Structured output is not currently supported with streaming. SinceDatabricksAdapteris streaming-only, the plan's original "sendresponse_formatinline on the streamed completion" (locked decision 3) cannot work on this endpoint. Fixed by making structured output a non-streaming completion (native structured output is supported that way), with the visible answer still streamed. This is the one substantive deviation from the approved plan, driven by real endpoint behavior.Scope
Non-streaming structured pass + a single
/chatfinal event (no progressive partial-JSON streaming). Schema is server-side (no client-sent JSON Schema). Code-config agents only — not markdownagent.mdagents or sub-agents.Notable choices to review
appkit.structured_output(internalAgentEventisstructured_output), matching theappkit.*siblings thatuse-agent-chat.tsswitches on.docs/docs/api/appkit/*.md) are intentionally not regenerated here —docs:buildpasses, but committing the regen pulls in unrelated pre-existing drift (e.g. feat(stream): make SSE maxEventSize configurable end-to-end #568'sstreamConfig). Left to the release regen.Verification
pnpm -r typecheck,pnpm check:fix,pnpm build,pnpm docs:build: all clean.output_parsed/structured_outputsurfaced) should be re-run against a live endpoint before marking ready-for-review.