feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5) by MarioCadenas · Pull Request #480 · databricks/appkit

MarioCadenas · 2026-07-16T14:26:33Z

Stack 4/5 · targets pr/agent-evals-3-datasets (review after #3).

CI-grade suite features.

Tool-call argument assertions: t.calledToolWith(name, expected) (deep-partial arg match).
Multi-turn dataset rows: userTurns(input) replays a row's full messages[] against one thread.
Concurrency: --concurrency N / maxConcurrency — bounded, order-preserving pool.
Pass-rate gating: --min-pass-rate 0..1 — gate on aggregate rate instead of all-must-pass.
Timeout / config / tags: enforce timeoutMs, load per-dir evals.config.ts, --tag filtering.
Retries: --retries N — re-run only on infra failures (turn/timeout), never on assertion failures.
Reporters: --reporter json|junit + --output for CI.

Includes two small dedup refactors (credential resolution; agent-dir listing / int parsing).

github-actions · 2026-08-09T08:14:51Z

This pull request has had no activity for 23 days and has been marked as stale. It will be closed in 7 days if there is no further activity. Add a comment, push a commit, or apply the no-stale label to keep it open.

github-actions · 2026-08-17T08:15:00Z

This pull request has been automatically closed because it had no activity for a month. Feel free to reopen it if you would like to continue the work.

github-actions · 2026-09-03T09:58:41Z

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

`@databricks/appkit` ⚠️ over budget

npm tarball (packed): 1.1 MB (+60 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	1.1 MB (+50 KB)	402 KB (+20 KB)
Type declarations	417 KB (+31 KB)	150 KB (+14 KB)
Source maps	2.2 MB (+102 KB)	754 KB (+40 KB)
Other	11 KB	3.7 KB
Total	3.8 MB (+183 KB)	1.3 MB (+74 KB)

Per-entry composition (own code — deps external (as shipped))

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
`.`	95 KB (-1 B)	2.5 KB	98 KB (-1 B)	external	313 KB
`./beta`	89 KB (+12 KB)	455 B (-1 B)	89 KB (+12 KB)	external	267 KB (+37 KB)
`./testing`	17 KB	0 B	17 KB	external	51 KB
`./tsdown`	520 B	0 B	520 B	external	813 B
`./type-generator`	23 KB	0 B	23 KB	external	65 KB

Chunks:

Entry	Chunk	Load	Size (gz)
`.`	`index.js`	initial	91 KB
`.`	`utils.js`	initial	4.0 KB
`.`	`remote-tunnel-manager.js`	lazy	2.5 KB
`./beta`	`beta.js`	initial	73 KB
`./beta`	`stream-manager.js`	initial	5.8 KB
`./beta`	`wide-event-emitter.js`	initial	3.2 KB
`./beta`	`databricks.js`	initial	3.2 KB
`./beta`	`configuration.js`	initial	2.2 KB
`./beta`	`service-context.js`	initial	1.3 KB
`./beta`	`client.js`	initial	434 B
`./beta`	`client-options.js`	initial	219 B
`./beta`	`supervisor-api.js`	lazy	192 B
`./beta`	`databricks.js`	lazy	141 B
`./beta`	`index.js`	lazy	122 B
`./testing`	`index.js`	initial	17 KB
`./tsdown`	`index.js`	initial	520 B
`./type-generator`	`index.js`	initial	23 KB

`@databricks/appkit-ui`

npm tarball (packed): 350 KB (-4 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	395 KB	132 KB
Type declarations	229 KB	84 KB (-1 B)
Source maps	766 KB	253 KB
CSS	16 KB	3.2 KB
Total	1.4 MB	473 KB (-1 B)

Per-entry composition (consumer bundle — deps bundled, peerDeps external)

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
`./js`	5.3 KB	49 KB	55 KB	208 KB	14 KB
`./js/beta`	20 B	0 B	20 B	0 B	0 B
`./react`	432 KB	49 KB	481 KB	1.3 MB	177 KB
`./react/beta`	1.0 KB	0 B	1.0 KB	0 B	1.9 KB

Chunks:

Entry	Chunk	Load	Size (gz)
`./js`	`index.js`	initial	5.2 KB
`./js`	`chunk`	initial	120 B
`./js`	`apache-arrow`	lazy	49 KB
`./js/beta`	`beta.js`	initial	20 B
`./react`	`index.js`	initial	430 KB
`./react`	`tslib`	initial	2.1 KB
`./react`	`apache-arrow`	lazy	49 KB
`./react/beta`	`beta.js`	initial	1.0 KB

⚠️ Over budget: a package's shipped tarball, or a browser entry's consumer bundle (deps included), grew by more than 5% (and >10 KB). This check will fail — reduce the size, or acknowledge the increase by updating bundle-size-baseline.json.

github-actions · 2026-09-03T10:02:48Z

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 33777821294 -R databricks/appkit -n appkit-template-0.71.0-pr.3b7e0a5-pr-agent-evals-4-suite-480 -D appkit-pr-480 \
  && unzip -o "appkit-pr-480/appkit-template-0.71.0-pr.3b7e0a5-pr-agent-evals-4-suite-480.zip" -d "appkit-pr-480" \
  && databricks apps init --template "appkit-pr-480"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

…al CLI Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

The eval driver captured tool-call names but discarded their arguments. Parse the function-call `arguments` JSON into `DriveResult.toolCallDetails` (the later `done` event's fuller args win over the initial `added`), and expose `t.calledToolWith(name, expected)` — passes when a call to `name` had args that deep-contain `expected` (nested-aware partial match; extra args ignored). Gate by default, like `calledTool`. `toolCalls: string[]` is unchanged. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

A managed-dataset row's `inputs.messages` can be a full conversation, not just one question. Add `userTurns(input)` to extract every user-message content in order; the example dataset eval replays them against one thread so the agent sees the accumulating conversation. Interleaved assistant/system turns are ignored — the agent generates its own responses. Single-user-turn rows are unchanged (one send). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Add `--min-pass-rate <0..1>` to the agent eval CLI: instead of requiring every eval to pass, exit non-zero only when the aggregate pass rate falls below the threshold. `summarize()` now returns `passRate` (passed / scored, excluding skips; 1 when nothing scored). Without the flag, behavior is unchanged (any gate failure fails the run). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Wire three eval fields that were declared but inert: - `timeoutMs` — runEval races the test against a per-eval timeout and records a clean non-passing result ("eval timed out after Nms") instead of hanging. Precedence: def.timeoutMs > runner/CLI --timeout > unbounded. Timer is always cleared. - `evals.config.ts` (defineEvalConfig) — discovered per-agent and loaded via the tsx loader; its maxConcurrency/timeoutMs apply as defaults (CLI flag > config > built-in). Judge model still comes from the CLI (needs creds the config lacks). - `tags` + `--tag <tag...>` — run only evals whose tags intersect the filter. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Add `--retries <n>` / `RunEvalsOptions.retries`: re-run an eval up to N extra times when it fails on an infrastructure error (a thrown error or timeout — `result.error` set), to absorb transient turn/stream flakiness. Assertion failures are never retried — a wrong reply is real signal, and retrying a flaky judge until it passes would corrupt the result. Each attempt gets a fresh driver. Extracted as `runWithRetries` (unit-tested for attempt counting, stop-on-success, and no-retry-on-assertion-failure). Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Add `--reporter <text|json|junit>` (default text) and `--output <file>` to the agent eval CLI. `formatResultsJson` emits `{summary, results}`; `formatResultsJUnit` emits a `<testsuite>` with a `<testcase>` per eval (`<failure>`/`<skipped>` as appropriate, all values XML-escaped). In json/junit mode the per-eval streaming is suppressed and human banners go to stderr so stdout stays clean for piping or `--output`; exit-code and pass-rate gating are unchanged. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

The discoverEvalConfigs test still built fixtures under config/agents/ from the pre-rename stack; the runner now discovers per-agent configs under server/agents/ (the folder-per-agent layout). Align the fixtures so the test matches the runner. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

MarioCadenas requested a review from a team as a code owner July 16, 2026 14:26

MarioCadenas requested review from ditadi and removed request for a team July 16, 2026 14:26

github-actions Bot added the stale label Aug 9, 2026

github-actions Bot closed this Aug 17, 2026

MarioCadenas reopened this Aug 17, 2026

github-actions Bot removed the stale label Aug 18, 2026

MarioCadenas force-pushed the pr/agent-evals-4-suite branch from 74a473e to 2dc29fe Compare September 1, 2026 16:32

MarioCadenas force-pushed the pr/agent-evals-4-suite branch from 2dc29fe to ed23b41 Compare September 1, 2026 17:27

MarioCadenas force-pushed the pr/agent-evals-4-suite branch from ed23b41 to 1e835b4 Compare September 2, 2026 09:03

MarioCadenas force-pushed the pr/agent-evals-4-suite branch from 1e835b4 to 637e9cd Compare September 3, 2026 09:56

MarioCadenas force-pushed the pr/agent-evals-4-suite branch from 637e9cd to 778effd Compare September 3, 2026 11:59

MarioCadenas force-pushed the pr/agent-evals-4-suite branch 2 times, most recently from 52f7f0b to d1b0765 Compare September 3, 2026 15:37

MarioCadenas added 6 commits September 3, 2026 18:14

refactor(appkit): dedupe databricks credential resolution in agent ev…

a4cbbb0

…al CLI Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

MarioCadenas added 2 commits September 3, 2026 18:16

MarioCadenas force-pushed the pr/agent-evals-4-suite branch from d1b0765 to 1478829 Compare September 3, 2026 16:18

Navigation Menu

Uh oh!

feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5) - #480

feat(appkit): eval suite features — concurrency, retries, reporters (stack 4/5)#480
MarioCadenas wants to merge 8 commits into
pr/agent-evals-3-datasetsfrom
pr/agent-evals-4-suite

MarioCadenas commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 17, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

MarioCadenas commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 9, 2026

Uh oh!

github-actions Bot commented Aug 17, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Bundle size report

@databricks/appkit ⚠️ over budget

@databricks/appkit-ui

Uh oh!

github-actions Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 AppKit PR bot

🔬 Run evals

📦 Try this PR's app template

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Sep 3, 2026 •

edited

Loading

`@databricks/appkit` ⚠️ over budget

`@databricks/appkit-ui`

github-actions Bot commented Sep 3, 2026 •

edited

Loading