Add deterministic AI reporter qualification gates

[Sean Larkin (TheLarkInn)]

Summary

• add a deterministic, network-free corpus for bootstrap/version, configuration, input, dependency-tool, operation, cache, network/auth, plugin, cancellation, and internal failures, plus successful controls
• compare AI, detailed plaintext, legacy, and full-detail file output with machine-readable blocking results and actionable per-case failures
• preserve stable diagnostic codes, root-cause ordering, classified context, remediation, payload-only stdout, warning suppression, and correlated absolute full-log references
• expose a qualification decision helper without wiring environment-based automatic reporter selection

Part of #5981.

Stack

• Parent/base: Add direct Rush reporter demo path #5997 (copilot/reporter-r5b-demo-reporters)
• Privacy prerequisite: Harden reporter telemetry privacy projection #5990
• Sibling exclusion: does not include the R7 branch

This PR does not copy or modify the telemetry projection from #5990. It applies the same producer-identity privacy boundary to presented reporter output and artifacts.

Corpus and gates

Gate	Blocking threshold	Result
Coverage	at least 10 failure cases and 2 successful controls	10 failures, 2 controls
Actionability	100% stable code, category, context, and remediation on failures	100%
Absolute size	at most 64 KiB AI output per case	976 bytes maximum
Context size	aggregate AI output at most 50% of legacy and plaintext	19.55% of legacy; 20.08% of plaintext
Determinism	identical normalized output across 3 runs	100%
Privacy	no classified secret or private producer/plugin identity leakage	100%
Full log	absolute, existing, owner-only where supported, complete, and failure-correlated	100%
Machine contracts	payload-only NDJSON plus warning suppression/detail behavior	100%

Corpus cases:

bootstrap-unsupported-node, configuration-invalid-json, input-unknown-project, dependency-package-manager, operation-build-failure, cache-restore-failure, network-auth-unauthorized, plugin-api-incompatible, logical-cancellation, internal-unexpected-error, success-no-warning, and success-warning-only.

External-service-shaped failures use stable canonical event fixtures. Returned results contain only case metadata, byte counts, booleans, and normalized hashes; absolute paths, private identities, and classified values are not stored.

Reproduce

cd libraries/reporter
rushx build
node scripts/runAiReporterQualification.js

The script prints the versioned machine-readable result and exits nonzero when any blocking gate fails.

Selection boundary

getQualifiedAiReporterDecision() validates built-in COPILOT_CLI detection and configured agentEnvironmentVariables, rejects absent or failed qualification results, and keeps RUSH_REPORTER=legacy authoritative.

It is intentionally not consumed by the Rush frontend in this PR. No environment variable automatically selects the AI reporter; the pre-major live path remains explicit --reporter or repository-experiment opt-in, and no-opt-in behavior remains legacy.

Validation

• rush build --to @microsoft/rush
• rushx _phase:test in libraries/reporter (314 tests)
• deterministic corpus, including 3 repeated normalized runs
• rushx _phase:test in libraries/rush-lib
• rushx _phase:test in apps/rush
• direct AI invalid-project failure: exit 1, payload-only NDJSON, actionable parser diagnostic
• no-opt-in COPILOT_CLI=1 legacy behavior and RUSH_REPORTER=legacy override
• rush check
• rush change --verify --no-fetch

Non-goals

• enabling environment-based automatic reporter selection
• changing Rush 6 or daemon-aligned defaults
• Heft child-reporter work
• bootstrap handoff integration
• telemetry projection changes

[Sean Larkin (TheLarkInn)]

Independent /pr-agentmerge review completed; fixes are in ac3fd75.

Findings fixed

• Full-log qualification previously inspected only the file reporter artifact. It now validates the AI-emitted absolute path, exact artifact correlation, format/completeness, existence, owner-only mode, required lifecycle content, external evidence, and failure/session/diagnostic correlation.
• Actionability now checks exact public context values and required privacy markers instead of key presence.
• The uniform 4 KiB synthetic blob was replaced by scenario-specific deterministic evidence. Aggregate ratios are supplemented by compact-case and per-case baseline gates so one oversized fixture cannot hide a regression.
• Local-sensitive producer identity is retained in the owner-only full log, while secret producer identity/values remain redacted. Secret diagnostics preserve honest aggregate counts and truncation without exposing code/category/detail or suppressing fallback errors.
• Future eligibility now requires both a passing qualification result and an explicit accepted privacy prerequisite. COPILOT_CLI/configured detection alone stays ineligible; RUSH_REPORTER=legacy remains authoritative. No telemetry implementation or Rush 6/default selection behavior was added.
• Added an adversarial reporter regression test: dropping artifactAvailable drives full-log to 0% with all affected case names in the CI diagnostic.

Corpus evidence

• 12 cases: 10 failures plus success and warning-only controls covering bootstrap/environment, configuration, input, dependency tool, operation/build, cache, network/auth, plugin, cancellation, internal, and success paths.
• Three independent standalone corpus executions were byte-identical across every normalized SHA-256.
• Gates: actionability/privacy/full-log/stdout/warnings/determinism all 100%; max AI case 976 B; compact max 969/2048 B; worst comparable per-case 68.47% of legacy and 72.00% of plaintext; aggregate 43.77% and 46.51%.
• Machine results were scanned for temp paths, fixture secrets, secret producer identity, and private plugin identity; none were present.

Validation

• Reporter build/lint/API Extractor and 319 tests passed.
• rush test --to @microsoft/rush passed the reporter, rush-lib, apps/rush, and dependency chain.
• rush check and stacked rush change --verify --target-branch origin/copilot/reporter-r5b-demo-reporters --no-fetch passed.

State / blockers

• PR remains based on Add direct Rush reporter demo path #5997, mergeable/clean, and auto-merge is disabled.
• No review threads existed to resolve.
• GitHub currently reports only the CLA check and no Actions run for this stacked head.
• Harden reporter telemetry privacy projection #5990 and Add direct Rush reporter demo path #5997 remain open; [rush reporter][R8] Qualify AI reporter auto-selection #5981 must remain open until both prerequisites merge, the qualification passes on main, and any approved activation remains gated by both prerequisites.

[Sean Larkin (TheLarkInn)]

Combined deep review of current head ac3fd75530.

1. HIGH local-sensitive messageEmitted text can be copied verbatim into AI stdout fallback output. Only public text may enter AI output. Use a redacted fallback and protected log reference otherwise.
2. LOW the oversized-record marker relabels non-public envelopes as public while retaining source and scope. Preserve privacy or remove those fields.

[Sean Larkin (TheLarkInn)]

Refresh CI follow-up published at 88dcf1a: the two negative qualification mutations now run in isolated, owned test workers with finite work/termination budgets and joined cleanup, rather than shared asynchronous prototype spies. A controlled6.2-second delay reproduced the timeout/next-test recursion before the correction and passed afterward.96 output/qualification/performance tests passed on Linux22 and actualWindows24/26; all15 canonical gate results are identical to the pre-correction results. Production code, APIs, corpus fixtures, measurements and quality thresholds are unchanged. Fresh exact-head CI 34429599947 is running; no failed old-head run was blindly retried.

[Sean Larkin (TheLarkInn)]

Verified refreshed R8 head88dcf1ab77df857e1e641ba9aa42f3a54600b91e: first current-head CI34429599947 passed all six complete Windows/Linux workflows, including package-manager, README, second-checkout and repeated-cache stages. The test-only mutation isolation correction preserves actual negative-gate detection while joining worker termination and cleanup; production reporters, APIs, corpus fixtures, paths, measurements and all quality thresholds are unchanged. This completes R8 refresh validation, not approval or authorization to merge/release.