Skip to content

Integrate negotiated Heft child reporting - #6000

Open
Sean Larkin (TheLarkInn) wants to merge 225 commits into
mainfrom
copilot/reporter-r7a-heft-reporting
Open

Integrate negotiated Heft child reporting#6000
Sean Larkin (TheLarkInn) wants to merge 225 commits into
mainfrom
copilot/reporter-r7a-heft-reporting

Conversation

@TheLarkInn

@TheLarkInn Sean Larkin (TheLarkInn) commented Aug 28, 2026

Copy link
Copy Markdown
Member

Part of #5980

Stack

Base: main. Parent #5997 and the rest of the Reporter core have landed.

The main-relative diff contains 45 R7 paths: the existing 44-path Heft slice plus its published watch-test cleanup fix. Landed R10 documentation is retained. Parallel R6 #5993 bootstrap work remains excluded.

Merge remains gated on current-head CI and independent approval.

Negotiated Heft reporting

Rush now allocates a pair of private inherited descriptors for operation children: one carries child-to-parent NDJSON events and the other returns the parent hello acknowledgement. The handshake negotiates protocol major, optional capabilities, required features, and parent-owned reporter context (reporter, log level, color, and terminal width).

Compatible Heft versions:

  • advertise heft-child-events-v1 and reporter-context-v1;
  • emit bounded externalOutput and structured diagnostics without selecting or constructing the parent's reporters;
  • preserve child session, parent request/session/operation correlation, privacy classification, and child sourceSequence;
  • propagate structured warnings back into Rush operation status calculation.

The Heft-side bootstrap uses a node-only frozen wire client rather than a runtime dependency on @rushstack/rush-reporter, avoiding a Heft/self-hosting dependency cycle. It activates only when Rush supplies both private descriptors; direct standalone Heft behavior is unchanged.

Fallback and problem matchers

Older Heft and opaque external children continue through ordered raw stdout/stderr. Missing or capability-incompatible negotiation falls back without losing output. Unsupported protocol majors or required features produce RUSH_PROTOCOL_UPDATE_REQUIRED; malformed, oversized, or truncated streams produce RUSH_PROTOCOL_INVALID_CHILD_STREAM.

Raw output is published before matcher diagnostics. Matchers consume ANSI-normalized copies keyed independently by operation, child record, and stdout/stderr stream, retain bounded partial lines, cap duplicates, and never mutate, suppress, or reorder the source output. They are disabled only after heft-child-events-v1 is actually accepted, preventing duplicate recovery when structured diagnostics exist.

Current landing qualification (September 11, 2026)

Published c6ed582fff9da05167aafd271e7be9348ea614f6 preserves the former R7 head and landed main@130daa8 as its two parents. Production/API and the joined watch-cleanup test are unchanged; only incoming R10 documentation/demo composition changes the old-side tree. Consumer build/types/lint/API and 343 focused Linux tests passed. Existing native evidence is not claimed as a new run. New hosted CI and human approval are still required.

Earlier validation (pre-landing snapshot)

  • rush build --only @rushstack/heft --only @rushstack/rush-reporter --only @microsoft/rush-lib --only @microsoft/rush --verbose
  • reporter protocol/matcher/Heft integration suite: 317 tests
  • Rush child-process cross-version fixtures: 5 tests
  • Rush operation stream/sharding/raw-order suite: 15 tests
  • Heft child bootstrap fixtures: 3 tests
  • apps/rush frontend suite: 28 tests
  • node apps/rush/src/test/sandbox/reporter-demo/run.mjs
  • rush check
  • rush change --verify --no-fetch

Coverage includes new Rush + new Heft, old/capability-less child fallback, unsupported major/required features, unknown optional and required events, malformed/truncated/oversized records, descriptor close and child crash, stdout/stderr interleaving, UTF-8 chunk limits, sourceSequence, request/session/operation correlation, privacy, matcher recovery and duplicate caps, sharded operations, warning status propagation, and parent context.

Non-goals

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Create the authoritative frontend reporter host before version selection, register global reporter controls, and preserve legacy output unless a non-legacy reporter is explicitly selected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Consume the repository experiment before Rush version selection, keep agent detection out of pre-major defaults, strip frontend-only controls before engine handoff, and preserve legacy verbosity compatibility.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn

Copy link
Copy Markdown
Member Author

Independent /pr-agentmerge review completed.

Findings fixed in 1f910c2

  • Structured Heft externalOutput bypassed Rush project logs and build-cache log chunks. Structured output now enters the archival pipeline through a no-reemit terminal provider, preserving child correlation while avoiding duplicate reporter output and problem matching.
  • Heft trusted inherited descriptor numbers and could write to or close unrelated descriptors when an intermediate launcher preserved environment variables but dropped fd mappings. Heft now requires pipe/socket descriptors before negotiation; Windows lifecycle commands explicitly retain safe raw stdout/stderr fallback because the shell launch path does not preserve Node fd mappings.
  • The parent host now rejects child-session changes, non-monotonic source sequence, malformed externalOutput payloads, and chunks above the 64 KiB protocol limit.
  • The reporter demo now accepts the valid fromCache operation terminal state when comparing rendered output with raw events.

Validation

  • rush test --only @rushstack/rush-reporter --only @microsoft/rush-lib --only @microsoft/rush: 6/6 operations passed.
  • Focused Heft HeftChildReporter suite: 4/4 passed.
  • Reporter demo passed, covering negotiated current-Heft events, parent correlation/sourceSequence order, chunk bounds, legacy rollback parity, raw grouping, and duplicate suppression.
  • rush check, rush change --verify, API Extractor, lint, and the focused post-fix review passed.
  • The full Heft suite reached four unrelated pre-existing GitUtilities.checkIgnoreAsync failures caused by this hidden worktree path; the changed Heft suite passes directly.

There are no review threads to resolve. #6000 remains stacked on open #5997 with auto-merge disabled. Merge is blocked until the ancestor chain reaches main, the PR is retargeted/rebased one step at a time with an R7-only diff, and full required CI/review gates pass.

@TheLarkInn

Copy link
Copy Markdown
Member Author

Combined deep review of current head 1f910c26e8.

  1. HIGH extra reporter descriptors are attached to every Unix shell operation. Forked grandchildren can retain the pipes and hang operation completion. Allocate descriptors only for negotiated children.
  2. MEDIUM the acknowledgement stream lacks an error handler. A closed child pipe can raise an unhandled EPIPE.
  3. MEDIUM forwarded child envelopes retain child-chosen source and privacy. Override both from trusted parent context before telemetry or reporter fan-out.
  4. MEDIUM handshake/context exceptions can escape the stream data handler and terminate Rush. Validate once and reject attachAsync() instead.

Stop reporter control scans at the pass-through separator and add an exactly-once frontend close contract across success, failure, and termination paths.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Keep reporter controls out of ts-command-line globals, gate incompatible engines before initialization, and enforce bounded signal and close-error behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Preserve unsupported custom reporter values until frontend ownership is unambiguous, and narrow emergency legacy stripping to the reporter selection flag.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Probe reporter ownership without requiring a value, then enforce strict reporter parsing only after frontend ownership is established.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
@TheLarkInn
Sean Larkin (TheLarkInn) force-pushed the copilot/reporter-r5b-demo-reporters branch from e30aa36 to 7159901 Compare August 28, 2026 20:08
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Track explicit operation graph iteration identities across lifecycle events and isolate reporter totals, diagnostics, and output spools by watch cycle.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d6318e80-5da9-4858-a147-817e8692f10e
Consume --verbose only for known actions that do not define it and parse repository opt-in value controls only when they are not command-owned. Preserve native aliases, declared custom values, unresolved plugin namespaces, and pass-through arguments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Retain the exact scoped producer API and WeakMap-backed plugin facades while preserving native-private parser/plugin members and real launch-boundary coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Document both the typed event sink and the frontend-assigned sessionId in the cross-version handoff without changing its shape.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve published early-failure, late-telemetry and operation-callback corrections; reconcile native lifecycle fields and telemetry references, with real branded parser regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Keep immutable errors intact, capture original pre-execution parser failures without changing legacy rendering, and observe final configured operation silence.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve the published R3C slice and review fixes while incorporating the authoritative server-rebased immediate parent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Retain the exact reviewed R5B slice, including unfiltered canonical full-detail logs and declaration-aware help.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Resolve physical cwd at parser entry so native Windows short names and directory aliases match Git repository paths. Keep real watch cancellation coverage and add symlink/junction regressions without mocking input analysis or watcher behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Native Node 24 and 26 validation showed that generic realpathSync and FileSystem.getRealPath retain 8.3 names. Use the existing native-realpath pattern to resolve the physical directory before configuration discovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Retain the exact 44-path Heft side slice and APIs while incorporating the qualified post-main core and native Windows watch fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Explicit pipes prevent Git line-ending notices from being mirrored onto the parent test stderr. Real setup failures still throw with the original captured error text. Reproduced the actual Rush production gate with process-local core.autocrlf=true and core.safecrlf=warn: unchanged tests exited with warnings before the fix and passed cleanly after it, without changing CI warning policy or watch assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve the existing slice and current parent histories; only capture successful fixture setup diagnostics, without changing production behavior, warning policy, or watch assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve the existing slice and current parent histories; only capture successful fixture setup diagnostics, without changing production behavior, warning policy, or watch assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Resolve both expected and actual link locations using native-backed realpath before comparing them. Add a real directory-alias regression that still rejects wrong and missing targets, and run it alongside the unchanged npm and Yarn integration workflows. Reproduced the previous lexical mismatch before the fix; the regression and complete suite pass under an invocation-owned aliased temporary root. No production code, dependency versions, or CI gates changed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve the existing slice and parent histories while carrying the validated test-helper correction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Preserve the existing slice and parent histories while carrying the validated test-helper correction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
Own the complete real watch test promise so Jest afterEach can cancel and join setup, parser execution, and finalization before releasing locks or resetting fixture files and mocks. Forward teardown cancellation only after production abort listeners are installed, and cover successful and failed finalization with deterministic gates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10
An error occurred while trying to automatically change base from copilot/reporter-r5b-demo-reporters to copilot/reporter-r5a-operation-adapter September 11, 2026 05:30
Retain the published R7 transport and owned watch-fixture changes while inheriting main 130daa8. Resolve squash-ancestry conflicts against the source-equivalent R5B tree and compose the R10 demo checks with the Heft negotiation and Windows fallback checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4747f826-8c83-495d-80df-3d1168662e10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Untrusted child diagnostics are insufficiently validated, allowing inconsistent metadata and invalid zero-based locations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 2 Medium severity

New issues introduced by this change (2)
Severity Finding
Medium severity libraries/​reporter/​src/​heft/​HeftDescriptorHost.ts — This accepts a known diagnostic code with an arbitrary category and summary/detail template.… View comment
Medium severity libraries/​reporter/​src/​heft/​HeftDescriptorHost.ts — The public diagnostic source contract defines line and column as 1-based, but this validator… View comment
What changed in this PR

Integrates negotiated structured reporting between Rush and Heft while preserving raw-output fallback and problem-matcher recovery.

Changes:

  • Adds bidirectional descriptor negotiation and parent-owned reporter context.
  • Streams correlated child output and diagnostics with validation and privacy controls.
  • Adds fallback, malformed-stream, watch-cleanup, and demo coverage.
File Description
libraries/​rush-lib/​src/​utilities/​Utilities.ts Supports custom child environment and stdio.
libraries/​rush-lib/​src/​pluginFramework/​RushSession.ts Exposes child reporter session context.
libraries/​rush-lib/​src/​logic/​operations/​test/​ShellOperationRunner.test.ts Tests Heft detection and reporter failures.
libraries/​rush-lib/​src/​logic/​operations/​test/​OperationGraphEventSink.test.ts Minor test formatting cleanup.
libraries/​rush-lib/​src/​logic/​operations/​test/​HeftChildProcessReporter.test.ts Covers negotiation, fallback, and corruption.
libraries/​rush-lib/​src/​logic/​operations/​ShellOperationRunner.ts Attaches reporter channels to Heft children.
libraries/​rush-lib/​src/​logic/​operations/​ReporterOperationEventSink.ts Integrates child reporting and problem matchers.
libraries/​rush-lib/​src/​logic/​operations/​OperationExecutionRecord.ts Adds structured child output routing.
libraries/​rush-lib/​src/​logic/​operations/​OperationEventSink.ts Defines child reporter sink contract.
libraries/​rush-lib/​src/​logic/​operations/​IOperationRunner.ts Extends operation runner context.
libraries/​rush-lib/​src/​logic/​operations/​HeftChildProcessReporter.ts Implements Rush-side descriptor host.
libraries/​rush-lib/​src/​index.ts Exports the internal reporter interface.
libraries/​rush-lib/​src/​cli/​test/​RushCommandLineParserReporterLifecycle.test.ts Makes watch-test cancellation cleanup reliable.
libraries/​reporter/​src/​test/​Telemetry.test.ts Updates protocol-version expectation.
libraries/​reporter/​src/​test/​Protocol.test.ts Tests capabilities, acknowledgements, and context.
libraries/​reporter/​src/​test/​ProblemMatchers.test.ts Tests incremental matcher behavior.
libraries/​reporter/​src/​test/​HeftIntegration.test.ts Expands protocol and security coverage.
libraries/​reporter/​src/​test/​Goldens.test.ts Updates forward-compatibility fixture.
libraries/​reporter/​src/​test/​__snapshots__/​Goldens.test.ts.snap Updates protocol snapshot.
libraries/​reporter/​src/​protocol/​ReporterProtocol.ts Advances protocol minor version.
libraries/​reporter/​src/​protocol/​ReporterHandshake.ts Adds capabilities, context, and ACK parsing.
libraries/​reporter/​src/​matchers/​ProblemMatcherRunner.ts Adds bounded streaming matching.
libraries/​reporter/​src/​index.ts Exports new reporter APIs.
libraries/​reporter/​src/​heft/​HeftDescriptorHost.ts Validates and correlates child events.
libraries/​reporter/​src/​heft/​HeftDescriptor.ts Allocates event and acknowledgement descriptors.
libraries/​reporter/​src/​heft/​HeftChildEmitter.ts Adds acknowledgement-gated child emission.
libraries/​reporter/​src/​events/​IReporterEventEnvelope.ts Adds parent-request correlation.
common/​reviews/​api/​rush-reporter.api.md Updates reporter API report.
common/​reviews/​api/​rush-lib.api.md Updates Rush API report.
common/​changes/​@rushstack/​rush-reporter/​copilot-reporter-r7a-heft-reporting_2026-08-28-08-40.json Records reporter minor change.
common/​changes/​@rushstack/​heft/​reporter-windows-pipes_2026-09-07.json Records Windows pipe fix.
common/​changes/​@rushstack/​heft/​copilot-reporter-r7a-heft-reporting_2026-08-28-08-40.json Records Heft reporting feature.
common/​changes/​@microsoft/​rush/​reporter-heft-review-fixes_2026-09-09.json Records stream-validation fixes.
common/​changes/​@microsoft/​rush/​copilot-reporter-r7a-heft-reporting_2026-08-28-08-40.json Records Rush integration change.
apps/​rush/​src/​test/​sandbox/​reporter-demo/​validateHeftOutput.mjs Validates demo child reporting.
apps/​rush/​src/​test/​sandbox/​reporter-demo/​run.mjs Adds Heft demo invocation.
apps/​rush/​src/​test/​sandbox/​reporter-demo/​README.md Documents platform behavior.
apps/​rush/​src/​test/​ReporterDemo.test.ts Tests demo validation rules.
apps/​rush/​src/​RushFrontend.ts Supplies child reporter context.
apps/​heft/​src/​pluginFramework/​logging/​ScopedLogger.ts Emits structured diagnostics when available.
apps/​heft/​src/​pluginFramework/​logging/​LoggingManager.ts Connects loggers to child reporting.
apps/​heft/​src/​pluginFramework/​logging/​HeftChildReporter.ts Implements frozen Heft wire client.
apps/​heft/​src/​pluginFramework/​logging/​HeftChildReporter.test.ts Tests negotiation and fallback.
apps/​heft/​src/​cli/​HeftCommandLineParser.ts Initializes negotiated child reporting.
apps/​heft/​src/​cli/​HeftActionRunner.ts Supports child reporter verbosity.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +109 to +116
typeof value.code !== 'string' ||
!isValidRushDiagnosticCode(value.code) ||
typeof value.category !== 'string' ||
value.category.length === 0 ||
(value.severity !== 'warning' && value.severity !== 'error') ||
typeof value.summaryKey !== 'string' ||
value.summaryKey.length === 0 ||
(value.detailKey !== undefined && typeof value.detailKey !== 'string') ||
Comment on lines +172 to +175
typeof value.source.file !== 'string' ||
(value.source.line !== undefined && !isNonNegativeInteger(value.source.line)) ||
(value.source.column !== undefined && !isNonNegativeInteger(value.source.column)) ||
(value.source.toolName !== undefined && typeof value.source.toolName !== 'string')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

2 participants