[ErenAri]

Draft. Merge only on an exact-head operator decision. Release-tooling fix: the readiness report enforces rules that written policy already states and the script did not. No new policy: no new threshold, no shortened window, no re-run policy chosen.

Hosted CI status (2026-10-19T18:47Z): the check runs on this head did not start. GitHub reports "The job was not started because your account is locked due to a billing issue." They are not code results; this head has no hosted CI result yet. All results below are LOCAL unless stated otherwise.

Head: fdae550d4d24c920f19e7f6f3a26290bcdc184d1 (pushed 2026-10-19; its independent review is in progress, so treat this head as under review). Previous head: 7f7a4effe2860cc4ee4219b5641d14635afc7b8d.

Defect (TD3)

Written policy and production-readiness-report.sh require the Falco expanded run to start after the candidate was built: run.started_at > the candidate evidence's generated_at (G). But the runner writes run.started_at when it assembles the report, after every target ran (internal/runner/runner.go:308-311).

In every preserved real report, run.started_at is later than the first target's start, for example:

• the upstream falcosecurity/libs report of run 38211321294: run 06:15:35Z, first target 06:12:36Z;
• the rc.4 LOCAL KubeArmor report: about 22 minutes later.

So a run that began before the candidate was built could be accepted as starting after it.

The consumer-evidence analysis found this; its reproduction table is in the task records.

Fix

• Start time: the Falco run's start is now the earliest of run.started_at and every target's started_at, and that must be after G.
• Missing start times: every required or passing target must record started_at, or the report is refused. An optional profile the runner never booted (unsupported, or fail in v0.3.7, with no started_at) does not count. This follows the independent review of 425335b; the expanded matrix has three such optional profiles.
• Manifest unchanged: it still names the report's own run.started_at, so its format does not change.

Target times are written in UTC by every VM backend (time.Now().UTC()), the format iso_epoch requires. The upstream v0.3.7 report carries them too.

The second commit makes this change and updates the runbook (docs/production-release-process.md, Falco bullet) to describe the rule. That bullet no longer says "as for the campaigns": campaigns are dated by their own metadata, and the review found that gap covered in practice by the campaign workflow's publication check.

Second defect (F3): drill and incident runs were never checked

Written policy (docs/production-release-process.md) says the rollback evidence is the evidence.md that rollback-drill.yml writes, and the incident note links the deliberately rejected promote-release run. The script checked only that each was a well-formed run ID.

A test specialist confirmed by reproduction that all of these were accepted (the first two with attestations verified):

• a rollback run of 1 (404 on GitHub);
• an incident linking 38446833676, a successful ci.yml pull-request run;
• a rollback completion in 2099 (reproduced in bypass mode).

Commits e033fe6 and 7f7a4ef fix this. With attestation verification on, both runs are looked up with gh api, and an unreadable run fails:

• Drill run: a successful, dispatched rollback-drill.yml run of this repository on main, started after the candidate was built, with the note's Completed: time inside the run.
• Incident run: a completed, dispatched promote-release.yml run of this repository that did not succeed.

Which rejection qualifies stays the operator's choice; a successful, unrelated or forked run never does. An old drill run is refused. An old rejected incident run is still accepted by design, because policy accepts an earlier candidate's incident. The test-only bypass is unchanged.

Third defect (F6, T5): canary runs were never looked up

Written policy says a canary is a run of the temporary release-candidate-canary workflow, the timed ones from its dated schedule, with components from that run, and that its plan job admits T+24h only 20-30 and T+72h only 68-78 whole hours after publication (T0), so that "any out-of-window invocation exits after the planning job" (docs/production-release-process.md:62-71, 180-191 at 7f7a4ef; .github/workflows/release-candidate-canary.yml:47-61). The readiness report made no gh call for canaries, even in attested mode. It checked only fields the evidence reports about itself.

Reproduced against 7f7a4ef in attested mode (all accepted):

• a canary naming a ci.yml run, a run on a branch, a run from a fork, a dispatched run labelled T+24h, a failed run, or a run that does not exist;
• a draft release, or a release published before G;
• a manual canary started before publication;
• a partial "Re-run failed jobs" of a failed in-window T+24h run, started after the window closed. A partial re-run does not run the plan job again: it keeps the first attempt's decision and writes new evidence later. So the runbook's claim that the plan job bounds every invocation was false for re-runs.

Commit fdae550 fixes this. With attestation verification on, readiness now:

• reads T0 from GET releases/tags/<candidate tag>. The release must exist, not be a draft, and have published_at at or after G;
• reads each canary run (GET actions/runs/<id>). It must be a completed, successful .github/workflows/release-candidate-canary.yml run of Kernel-Guard/bpfcompat, from workflow_dispatch (manual) or schedule (timed). Timed runs must be on main;
• reads each attempt (GET .../attempts/<n>, n = 1 to run_attempt) and selects the successful attempt whose [run_started_at, updated_at] contains the evidence's completed_at. A failed attempt never counts;
• reads that attempt's jobs (GET .../attempts/<n>/jobs). Its Assemble RC canary evidence job must have succeeded and started at or after the attempt's run_started_at. This proves the job ran in that attempt and was not carried over or skipped: an out-of-window or late "Re-run all jobs" attempt also concludes success, with every job skipped;
• requires that attempt to start at or after T0;
• for T+24h and T+72h, refuses an attempt that started 31h (79h) or more after T0. This is the plan job's own bound (whole hours 20-30 and 68-78). It applies to every attempt;
• for a first attempt, which ran its own plan job, also requires completed_at at or after T0+20h (T0+68h).

The output records workflow_run_attempt, attempt_started_at and head_branch for each canary, and candidate.published_at. The Markdown table gains Attempt, Attempt started and Branch columns. Bypass output records them as null / "not looked up". The new keys are additive; the stable gate reads only .canaries | length == 3.

What this does not choose (open operator decision, D-CAN-2): a re-run that started inside the window is accepted, because that is what the windows as written imply. The alternatives stay with the operator:

• count no re-run (first attempts only): a tightening, and a one-line change;
• also count a late re-run whose checks all ran in the window, or any re-run of an in-window run: relaxations, which need the runbook text changed first.

This should be answered before the rc.5 T+24h window opens (T0+20h): under "no re-runs", an in-window re-run is futile.

Two readings, recorded rather than enforced:

• Manual canary branch. No written rule names the branch of the manual dispatch. The runbook names the workflow and the timing only (docs/production-release-process.md:62-71). The branch-shakeout rule in docs/engineering/orchestration.md:79 covers three other workflows. So head_branch is recorded for the manual canary, not enforced. Requiring main is a one-condition change if the operator wants it.
• Evidence job by display name. The jobs API exposes only a job's display name, not its YAML key. The workflow names its evidence job Assemble RC canary evidence (release-candidate-canary.yml:388-389). A renamed job fails closed. The test checks that, while the temporary workflow exists, it still uses that name.

The runbook (docs/production-release-process.md) now says that a partial re-run is not a new invocation and that the plan job alone does not bound it. It no longer says "The plan job itself enforces the upper bounds", and it describes the new checks and the open decision. The test's gh stub answers the release, runs, attempts and jobs in the same commit as the script change. The test runs in ci.yml and in the release quality gates of every main push and v* tag build, so it must be green at every commit.

Validation

Evidence labels: LOCAL = run on the maintainer host; hosted = GitHub Actions. PR CI is not qualification.

• Hosted, previous head 7f7a4ef: 12 of 12 check runs succeeded (re-read by GET 2026-10-19T18:23Z). Hosted CI for this head has not run (account billing lock; see the note at the top).
• LOCAL, fdae550: scripts/production-readiness-report_test.sh PASS. It now has 26 new canary cases (22 refusals, 4 acceptances) plus lookup and record assertions:
  • refused: another workflow, not on main (T+72h), fork, dispatched T+24h, scheduled manual, failed run, nonexistent run (404);
  • refused: draft release, unreadable release, release published before G, manual attempt started before T0;
  • refused: completion outside the run, a failed attempt whose span contains completed_at, a late re-run whose completed_at was edited into the window;
  • refused: skipped evidence job, missing evidence job, evidence job carried over into a re-run attempt;
  • refused: first attempt started after T0+31h, late partial re-run (attempt 2 started after T0+31h), T+72h re-run started exactly at T0+79h, first-attempt completion in [G+20h, T0+20h) and in [G+68h, T0+68h);
  • accepted and recorded: an in-window re-run (attempt 2), a re-run one second before the window closes, evidence from an earlier successful attempt when a later full re-run skipped every job, a manual canary dispatched on a branch (branch recorded);
  • every lookup is made, attempts, starts, branches and T0 are recorded, and bypass output claims none of them.
• LOCAL RED: the same test against unmodified 7f7a4ef fails at the first new assertion. In a report-mode copy, all 39 new assertions fail: the 26 cases, the 10 lookups, the 2 record checks and the re-run Markdown row.
• LOCAL mutation check: 28 mutants, each one asserted single-match substitution, each killed by the full test. One rule is removed per mutant: event, path, head_branch, repository, run conclusion, attempt selection by time, attempt success filter, latest-attempt-only, evidence-job check (whole, conclusion, in-attempt start), attempt start ≥ T0, upper bound (removed, attempt-1-only, < to <=, 31→32, 79→80), attempt-1 floor (removed, 20→19, 68→67), T0 ≥ G, release not draft, unreadable run or release tolerated, attempt fields not recorded, manual branch enforced, and the workflow renaming its evidence job.
• LOCAL real-data replay: the API answers of the six real rc.3 and rc.4 canary runs and their releases, read by GET, served to the attested path. Both candidates are accepted, all at attempt 1. The real runs are 30560118663, 30649359059, 30756195754, 38609546766, 38802817455 and 39063641609. completed_at was approximated by the real evidence step's start, because the artifacts were not downloaded. On real data, the margin between completed_at and the attempt's updated_at is 3-8 s, and between the attempt's start and the evidence job's start, minutes.
• LOCAL: the full local replay of ci.yml's test and build steps on fdae550: 40 steps ran, 0 failed, 1 skipped (go mod download, by design of the vendored offline replay); shellcheck clean at warning severity, and at default severity apart from one pre-existing SC2015 info; a trial merge with release: let the stable PR and tag build pass the stable canary-pin and doc-pin checks #266 (4cb4516) merges cleanly, and the test passes on the merge.
• Earlier commits (unchanged):
  • the positive Falco case shaped like a real report and its two refusals;
  • independent reviews of 425335b and e033fe6 (no blocker; their should-fixes are addressed);
  • eleven drill and incident refusals in attested mode;
  • shellcheck, make check-release-consistency and the freeze check.

Not changed (verified, left as decisions or residuals)

• D-CAN-2 (policy, operator): whether a re-run of a timed canary counts. Readiness applies the windows as written (a re-run started in the window counts); tightening or relaxing is the operator's call.
• Canary run binding to the candidate: the run is tied to its evidence by workflow, event, branch, repository, attempt and time, not by which candidate the run tested. A real successful canary run started after T0 but before the rc.5 designation merged (so testing rc.4) could be named by an assembled rc.5 record. Binding the evidence artifact's digest, or checking that the canary workflow at the run's head_sha names the candidate, is a possible hardening.
• T0 source: readiness uses the release's published_at; the plan job uses the designation's CANDIDATE_PUBLISHED_AT. The window rules are exact necessary conditions only if the two are equal. Only release: prepare v0.4.0-rc.5 metadata and an offline, AST-based RC preflight (replaces #264) #265's read-only preflight compares them.
• Clock skew: completed_at (runner clock) must fall inside the attempt span (GitHub clock), with no tolerance. A runner clock running several seconds fast could refuse a genuine canary. This fails closed, like the drill check.
• GitHub semantics relied on: an earlier attempt's updated_at stays at its own end after a re-run (observed on run 38446833676; not documented). Re-run behaviour itself is from GitHub docs and one observation; no canary run has ever been re-run.
• Canary components other than the five <check>.json records are still not checked on disk, yet components_verified: true is written (READINESS-CLAIMS F6-C6). The external-consumer run is not looked up.
• Stable gate (release-artifacts.yml, not changed here): it does not require the new fields, so a readiness file made with an old checkout would still pass it. The readiness rollback section check (R07 F1) is being handled on a separate branch.
• Other copies of the inaccurate claim: the canary workflow's header comment (release-candidate-canary.yml:3-5, corrected in the rc.5 designation PR) and the preflight comment in release: prepare v0.4.0-rc.5 metadata and an offline, AST-based RC preflight (replaces #264) #265 (scripts/rc-preflight.sh:33-34).
• Run binding (drill): the drill note is tied to its run by time, repository and event, not by its alias or digests (review 3, nit 1). A note naming another candidate's digests under a real drill run of this repository in the right time window would pass. Binding the run's artifact digest is a possible hardening.
• Re-runs and clock skew (drill): a note from attempt 1 of a later re-run drill is refused, because the run's start resets on re-run; a runner clock running fast can refuse a genuine note. Both fail closed.
• incident.candidate_bound: a text match (F2), informational only.
• Campaign start times: they use the report's assembly time too; the campaign workflow's publication check covers this in practice.
• The Falco workflow_run_id and commit_sha remain operator-asserted.
• Whether the run executed the candidate's validator is still reported (candidate_bound), not enforced.
• No CHANGELOG entry, to avoid conflicting with release: prepare v0.4.0-rc.5 metadata and an offline, AST-based RC preflight (replaces #264) #265. That PR's rc.5 bullet already states the Falco rule; proposed lines for the canary lookups are in the task record.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}