Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 41 additions & 13 deletions docs/design/2026-11-14-merge-honesty.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

[cols="1,3"]
|===
|Status |PROPOSED. Not ruled. No code changes with this note.
|Status |RULED by the owner on 2026-11-14. See <<rulings>>. No code changes with this note.
|Date |2026-11-14
|Asks |5 decisions from the owner, in <<decisions>>.
|Authority |The owner asked for this note on 2026-11-14, before any work order changes.
Expand Down Expand Up @@ -72,8 +72,9 @@ category.
qualified failure.

|`lib/neural/radial_neural_network.ex`, `detect_novelty/2` (line 145)
|A network that is not trained returns `:untrained`. The gate reads that as
"not novel".
|A network that is not trained returns `:untrained`. The coordinator then
sets `is_novel` to `false` (`lib/neural/coordinator.ex`, line 765), and the
dispatch guard reads that field.
|The one gate for unknown findings is open until training.

|`lib/neural/variational_autoencoder.ex` (lines 80 to 84)
Expand All @@ -92,8 +93,9 @@ things become 1, and no record says so.
The dispatch tier of a recipe follows from its confidence and its annealing
stage. With a merged subject, a recipe can reach `auto_execute` (0.95 or more)
on the successes of findings that only look like the finding in front of it.
This is the reverse of acceptance A3 of issue 984, which asks that each
learning update rest on evidence for that update.
Acceptance A3 of issue 984 does not forbid this: it asks that no learning
rests on unverified evidence, and a verified outcome under a wide subject
meets it. The risk here is a separate one, about the subject of the evidence.

echo-types gives 2 proved results that bound what a later repair can do:

Expand All @@ -113,9 +115,13 @@ Each outcome row, and each receipt of `E0-08`, records the pattern id and the
kind of match (`exact`, `category`, `fuzzy`). The confidence stays one value
for each recipe.

* Result: each merge becomes `Fused(τ)`. An audit can split the outcomes of a
recipe by pattern. Nothing in the dispatch changes.
* Cost: small. 2 fields, in the one place that writes the row.
* Result: a merge by the recipe matcher becomes `Fused(τ)`. An audit can
split the outcomes of a recipe by pattern. Nothing in the dispatch changes.
* Limit: 2 findings that share 1 pattern id stay merged. The label cannot
tell them apart until the pattern identity changes (decision 3).
* Cost: medium. The matcher returns the recipe only, and the router carries
no kind of match. So the 2 fields must go from the matcher through the
router to the place that writes the row.
* Limit: the risk stays. A recipe can still reach `auto_execute` on merged
evidence. The record only makes it visible.

Expand All @@ -138,8 +144,11 @@ Do option A in the E0 work. Then measure, from the labelled rows, how often a
recipe gets outcomes from more than 1 pattern and how far their success rates
differ. Rule on option B with those numbers.

* Result: no evidence is lost from now on, and the larger change rests on a
measurement, not on a guess.
* Result: the pattern and the kind of match are kept from now on, and the
larger change rests on a measurement, not on a guess.
* Needs: the fix check by finding (the second plain defect) before the
measurement. With the check of today, a rate for a pattern also measures
how often a category occurs twice in a repository.
* Cost: the risk of option A stays until the ruling on B.

=== Option D: no change
Expand Down Expand Up @@ -168,6 +177,25 @@ the direct choice is on purpose, for that reason.
the ruling?
. The novelty gate. Does `:untrained` mean "novel" (fail closed) from now on?

[[rulings]]
== Rulings

The owner ruled on 2026-11-14, through the selector.

[cols="1,4,2",options="header"]
|===
|Decision |Ruling |Work order

|1 |Option C: label the merge now, and rule on the split after a measurement. |`E0-18`
|2 |A `category` match or a `fuzzy` match reaches `review` at most. |`E0-19`
|3 |A new key that does not depend on wording, as its own design work order. |`E0-20`
|4 |3 small work orders now for the 3 plain defects. |`E0-21`, `E0-22`, `E0-23`
|5 |An untrained novelty network means novel (fail closed). |`E0-24`
|===

The measurement of option C has no work order yet. It needs `E0-18` and
`E0-22` first.

== What this note does not propose

* No new way to decide whether 2 findings are alike. The trope repositories
Expand All @@ -191,6 +219,6 @@ the direct choice is on purpose, for that reason.

== When to update this memo

When the owner rules on <<decisions>>, set the status and record each ruling
here. When a work order shows that an "inferred" result is false, correct the
table.
When the measurement of option C exists, record it here and ask the owner
for the ruling on option B. When a work order shows that an "inferred"
result is false, correct the table.
18 changes: 17 additions & 1 deletion docs/operations/e0-closeout/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,17 @@ function name is the stable locator. Measure again before you edit.
|E0-15 |Truth update of the documents |S |sonnet |$3 |none
|E0-16 |E0 acceptance run and receipt |M |opus |$10 |all
|E0-17 |The test suite does not write into the fleet checkout |S |sonnet |$3 |none
|E0-18 |Each outcome names its pattern and its kind of match |M |sonnet |$10 |none
|E0-19 |A merged match reaches review at most |S |sonnet |$5 |E0-18
|E0-20 |The pattern identity: design and ruling packet |Design |opus |$10 |none
|E0-21 |A fuzzy match ranks by its hits |S |sonnet |$4 |none
|E0-22 |A fix is checked against its own finding |M |sonnet |$8 |none
|E0-23 |A placeholder network adds no confidence |S |sonnet |$4 |none
|E0-24 |An untrained novelty network means novel |S |sonnet |$5 |none
|===

The caps add up to $151. The estimate for the whole close-out, with rework, is
The caps of `E0-01` to `E0-17` add up to $151. The estimate for that
close-out, with rework, is
$115 to $315. That figure is an estimate. The first 4 runs (`E0-17`, `E0-06`,
`E0-01`, `E0-13`) are the test run that replaces it with measured numbers.

Expand All @@ -124,6 +132,11 @@ and the headless profile of this kit are in hypatia only, because the owner
chose hypatia first on 2026-11-14. Put them in `gitbot-fleet` before those 2
runs, or run those 2 work orders in a session with a person.

`E0-18` to `E0-24` come from the owner rulings of 2026-11-14 on
`docs/design/2026-11-14-merge-honesty.adoc`. They are not part of the 7
acceptance statements of issue 984. Their caps add up to $46, so all 24
caps add up to $197.

=== Order

. First of all: `E0-17`. Until it merges, each `mix test` adds test rows to
Expand All @@ -133,6 +146,9 @@ runs, or run those 2 work orders in a session with a person.
. Second lane, each starts with a design note for the owner: `E0-04`, `E0-07`, `E0-08`.
. After the rulings: `E0-05` (after `E0-04`), `E0-02`, `E0-09`, `E0-10`, `E0-11`, `E0-12`.
. Last: `E0-16`.
. Beside the lanes, in any order: `E0-21`, `E0-22`, `E0-23`, `E0-24`,
`E0-18`, then `E0-19`. `E0-20` gives the owner a design note. No one
measures a success rate for each pattern before `E0-18` and `E0-22` merge.

== Rules for every run

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,26 @@ Commit `4d7f059`, read on 2026-11-14.
* A fleet row has the identity `fleet_import_id` (`lib/outcome_tracker.ex`, line 55).
A local row has only a provisional identity (file and line).

== Theory inputs

Read on 2026-11-14 at echo-types `dedda75` and residual-evidence-types
`6f94640`. Each file has `--safe --without-K --cubical-compatible`. No prover
was run for this work order. Use these results as design rules. A use in
Hypatia is a transcription, and its match to the Agda is assumed, not proved.

* Apply a withdrawal to the log, then compute again. Do not correct the
stored confidence. An aggregation that merges 2 inputs has no canonical
inverse (echo-types `EchoAggregation.agda`, line 210), and a filter
commutes with a summary only when the summary keeps what the filter reads
(`EchoSelectiveProjection.agda`, line 106). The count does not keep the row.
* Check each row against the log that remains, in 1 place. Survival is proved
against the retracted context, never against the full one
(residual-evidence-types `Revision.agda`, lines 180 to 191).
* A stable id for a local row is a need, not a choice: exact recovery forces
distinct residues for distinct inputs (`EchoFiniteResidue.agda`, line 36).
* Not proved anywhere: that retraction is idempotent or free of order, and
any statement on confidence. A withdrawal of a failure row raises confidence.

== Steps

. Measure the facts again. Stop if one no longer holds.
Expand Down
20 changes: 20 additions & 0 deletions docs/operations/e0-closeout/work-orders/E0-08-receipt-design.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,26 @@ Commit `4d7f059`, read on 2026-11-14.
* Estate rules: JSON is I-JSON in JCS form, and a new identifier follows
ADR-008 (see `.claude/CLAUDE.md` of `metadatastician/standards`).

== Theory inputs

Read on 2026-11-14 at echo-types `dedda75` and residual-evidence-types
`6f94640`. Each file has `--safe --without-K --cubical-compatible`. No prover
was run for this work order. Use these results as design rules. A use in
Hypatia is a transcription, and its match to the Agda is assumed, not proved.

* The receipt names each row of evidence by a stable id. It does not name a
count: a count cannot give back its evidence (echo-types
`EchoFiniteResidue.agda`, lines 36 and 47).
* The receipt is an index into the log that is kept. It is not a record that
can undo an update by itself.
* `missing_evidence` is a set of gaps. A later revision fills a gap and never
overwrites a field that is present (`EchoPartialSection.agda`, line 162,
`fill-guarded`). A withdrawal removes information, so it is not a fill: it
needs its own record.
* By the owner ruling of 2026-11-14 on
`docs/design/2026-11-14-merge-honesty.adoc`, the receipt also names the
pattern and the kind of match (`exact`, `category`, `fuzzy`). See `E0-18`.
Comment on lines +62 to +64

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make receipt emission depend on E0-18

This newly requires receipts to name the match kind, but E0-09 still depends only on E0-05 and E0-08, while the README schedules E0-18 independently “in any order.” If E0-09 runs first, none of its three learning entry points receives the dispatch match kind, so it must either stop under its unfillable-field condition or emit receipts without the owner-required label; no later work order backfills those receipts. Add E0-18 as a dependency of E0-09 and order it before receipt emission.

Useful? React with 👍 / 👎.


== Steps

. Measure the facts again. Stop if one no longer holds.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,25 @@ Commit `4d7f059`. The run facts are from 2026-11-14 with Idris 2 version 0.7.0.
* `verification/proof-ledger.jsonl` has one row for each proved obligation.
`verification/PROOF-STATUS.adoc` is rendered from it.

== Theory inputs

Read on 2026-11-14 at echo-types `dedda75` and residual-evidence-types
`6f94640`. Each file has `--safe --without-K --cubical-compatible`. No prover
was run for this work order. Use these results as design rules. A use in
Hypatia is a transcription, and its match to the Agda is assumed, not proved.

Candidate properties for the model, beside those of the design:

* The log splits exactly into the withdrawn row and the rest, and joins
again (residual-evidence-types `Revision.agda`, lines 101 to 141). Hypatia's
`Retraction.idr` does not have this yet.
* A receipt survives a withdrawal only when each row that it names is in the
log that remains (same file, lines 180 to 191).
* A revision of a receipt only fills gaps (echo-types
`EchoPartialSection.agda`, line 162).
* A known gap, not to hide: `Retraction.idr` retracts by position, and the
run time retracts by name. State which of the 2 the model proves.

== Steps

. Confirm that `E0-09` is merged. Measure the facts again.
Expand Down
64 changes: 64 additions & 0 deletions docs/operations/e0-closeout/work-orders/E0-18-merge-label.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
// SPDX-License-Identifier: CC-BY-SA-4.0
// Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
= E0-18: Each outcome names its pattern and its kind of match
:wo-id: E0-18
:wo-repo: hyperpolymath/hypatia
:wo-size: medium
:wo-agent: e0-executor
:wo-model: sonnet
:wo-effort: medium
:wo-cap-usd: 10
:wo-max-turns: 200
:wo-branch: feat/e0-18-merge-label
:wo-depends: none

== Goal

Each outcome row records the pattern that it is about and the kind of match
that chose its recipe: `exact`, `category` or `fuzzy`. A merge of 2
findings under 1 recipe then has a label.

== Authority

Owner ruling of 2026-11-14 on `docs/design/2026-11-14-merge-honesty.adoc`, decision 1 (option C: label now, split after a measurement).

== Facts measured

Commit `b660d08`, read on 2026-11-14. The cited files have no change up to `3b8551c`.

* `lib/recipe_matcher.ex`, `best_recipe_for_pattern/2` (line 123) tries an
exact match, then `category_match_recipe/2` (line 142), then
`fuzzy_match_recipe/2` (line 156). It returns the recipe only. The caller
cannot see which of the 3 matched.
* `lib/triangle_router.ex`, `route/3` (line 35) carries no kind of match.
* `lib/outcome_tracker.ex`, `persist_outcome/6` (line 116) writes
`"pattern_id" => nil` (line 122).
* `test/recipe_matcher_test.exs` has the tag `:verisim_data`, so CI does not run it.

== Steps

. Measure the facts again. Stop if one no longer holds.
. Write the tests first, in a new test file that has no `:verisim_data` tag
and that uses recipes of its own in a temporary folder.
. Add a function beside `best_recipe_for_pattern/2` that returns the recipe
and the kind of match. Keep the old function and its result, so that no
caller breaks.
. Carry the pattern id and the kind of match from the router to the outcome
row through the `metadata` map of `record_outcome/5`. Write them as
`"pattern_id"` and `"match_kind"`.
Comment on lines +46 to +48

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Carry match kind across the dispatch-runner boundary

The production route and outcome write are not connected in-process: DispatchManifest serializes the routed action, then gitbot-fleet's runner later invokes mix hypatia.record_outcome. The manifest and CLI currently have no match_kind handoff, so carrying it only through the router and record_outcome/5 metadata can satisfy a local test while real outcome rows still receive "unknown". E0-18 must include the manifest, CLI, and gitbot-fleet runner transport (or add a dependent fleet work order) so it records the kind chosen for the actual dispatch.

Useful? React with 👍 / 👎.

. Do not change confidence, the annealing stage or a tier in this work order.

== Acceptance

* New tests show the kind `exact`, `category` and `fuzzy`, 1 test for each.
* A test shows a row with the pattern id and the kind of match of its dispatch.
* A row from a caller that gives no kind has `"match_kind" => "unknown"`. It
is not written as `exact`.
* `mix test` has 0 failures. No file that was formatted before the change is unformatted after it.

== Stop and ask

* A path writes an outcome and has no pattern at that point. List each such
path in the pull request body. Do not invent a pattern id.
* The change needs a new field in the fleet outcome file of `gitbot-fleet`.
* The cost reaches the cap before the acceptance checks pass.
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// SPDX-License-Identifier: CC-BY-SA-4.0
// Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
= E0-19: A merged match reaches review at most
:wo-id: E0-19
:wo-repo: hyperpolymath/hypatia
:wo-size: small
:wo-agent: e0-executor
:wo-model: sonnet
:wo-effort: medium
:wo-cap-usd: 5
:wo-max-turns: 120
:wo-branch: fix/e0-19-merged-match-tier
:wo-depends: E0-18

== Goal

A recipe that a `category` match or a `fuzzy` match chose never runs with
no person. Its highest tier is `review`. Only an `exact` match can reach
`auto_execute`.

== Authority

Owner ruling of 2026-11-14 on `docs/design/2026-11-14-merge-honesty.adoc`, decision 2.

== Facts measured

Commit `b660d08`, read on 2026-11-14.

* `lib/neural/dispatch_guard.ex`, `enforce/2` (line 69) already lowers an
action for a novel finding.
* The tiers are: `auto_execute` from confidence 0.95, `review` from 0.85,
`report_only` below (`.claude/CLAUDE.md`, design constants).
* `E0-18` gives each dispatch its kind of match.

== Steps

. Confirm that `E0-18` is merged. Stop if it is not.
. Measure the facts again. Find the 1 place where the tier of a dispatch is
decided, and name it in the pull request body.
. Write the tests first.
. Apply the cap there. An unknown kind of match gets the cap too.

== Acceptance

* Tests show: confidence 0.99 with `exact` gives `auto_execute`; confidence
0.99 with `category`, with `fuzzy` and with `unknown` gives `review`.
* A test shows that the cap never raises a tier.
* `mix test` has 0 failures. No file that was formatted before the change is unformatted after it.

== Stop and ask

* The tier is decided in more than 1 place and the places disagree.
* The cost reaches the cap before the acceptance checks pass.
Loading
Loading