Skip to content

test(qa): hold the DOCX fidelity corpus to its baseline whichever way a line moved - #976

Merged
DemchaAV merged 3 commits into
2.5-devfrom
fix/docx-fidelity-gate-both-directions
Oct 29, 2026
Merged

DemchaAV merged 3 commits into
2.5-devfrom
fix/docx-fidelity-gate-both-directions

Conversation

@DemchaAV

@DemchaAV DemchaAV commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Why

The DOCX fidelity gate only ever looked for documents moving away from the page. FidelityBaseline.regressions failed a line that stood more than 0.5pt further from the engine's PDF than its baseline held it, a line the baseline found and the editor no longer sets at those words, and a page count further from the engine's. Measured on the gate as it stood, by feeding it baselines and measurements that differ in one way each:

step old gate
+1.00 -> +1.60pt (further, past the slack) fails
+1.00 -> +0.49pt (0.51pt nearer) silent
+1.10 -> +0.00pt (1.10pt nearer — what #944 and #948 did to 227 lines) silent
+5.00 -> +0.00pt (five points nearer) silent
+3.00 -> -3.00pt (six points of movement, same distance) silent
editor on 2 pages → 1, the engine's own count silent
a line found that the baseline does not hold silent
a line more drawn on the page, the same ones found silent

The fifth row is the one that shows the rule was wrong rather than merely narrow: written as "is the new drift further from the page than the old one", it cannot see a line that crosses the page's own line at the same distance, however far it travels — +3.00 -> -3.00pt is six points of movement and both values stand three points away.

And silence about a move nearer the page is not tidiness. Unwritten, the baseline keeps the further value, so a later change may walk the line back to it and spend the 0.5pt of slack on top of that.

Held one-sidedly, the Word baseline went stale in 1609 of its 5165 rows across 74 commits without a single red run: 1388 rows moved down the page — 717 nearer it, 635 further, and 36 crossing it to stand the same distance on the other side — and 229 moved across, 228 of them nearer by as much as 1.10pt. 228 of those rows had moved more than the slack (226 across and 2 down, both of the latter crossing the page's line). And yet no row's distance from the page had grown by more than 0.48pt — 21 rows of cv-serif_headline, cv-panel and invoice-obsidian by that much, the rest by 0.36pt or less — which against half a point of slack is exactly why a rule reading distances stayed quiet. Those same three documents are the ones that ended 0.48pt further, where the next change to touch them would have failed the gate for drift it did not cause.

What changed

FidelityBaseline.regressions becomes divergences, and holds every measurement as a step from where the baseline puts it rather than as a distance to compare:

  • a found line fails when its drift moves more than LINE_SLACK (0.5pt), down or across, nearer the page or further — Math.abs(now - was), not Math.abs(now) against Math.abs(was);
  • a document fails when the editor sets it on another number of pages, in either direction, and when the engine draws it on another number of pages or another number of lines;
  • a document fails when the set of its found lines is not the set the baseline holds — a line lost, as before, and now a line found that the baseline does not hold;
  • a line the baseline holds a place across for, measured without one, fails rather than passing unheld;
  • a baseline row whose drift, place across, median or p90 is not a number is refused on read, and a measurement that carries no number on either axis fails. Every comparison against NaN is false — Math.round(NaN) is 0 — so such a row held its line to nothing however far it moved: a line 500pt out of place produced no divergence at all. These files are patched by hand, so one mistyped cell was enough;
  • a rewrite now keeps the notes the file it replaces carried, by either path — a -Dupdate=true run and the CI artifact that gets copied over the Linux baseline. Those notes are where a row's reckoning is written down, and a rewrite is now the ordinary answer to a document moving, so dropping them would spend the record this gate exists to keep. The header is known by standing first, since it cannot be told from a note by its text, and a baseline that does not open with it is refused rather than half-read.

Each failure names the step and its difference, "aa" +1.10 -> +0.00pt (-1.10), so the direction is in the message rather than left for the reader to work out. The remedy is in the assertion itself: a change that moves documents — nearer the page included — rewrites the baseline in the same commit with -Dgraphcompose.docxFidelity.update=true, and that diff is the record of what moved and which way.

Two things deliberately left alone:

  • One slack for both directions. 0.5pt is rounding, and rounding is the same size whichever way a line rounds; nothing measured here argues for a second number.
  • The row aggregates stay ungated. A row's median, p90 and lines past 2pt follow from the lines, which are now held one by one, and an average is the wrong place to catch a tail.

Documentation that stated the old rule is corrected: two paragraphs of CONTRIBUTING.md, the DOCX Fidelity job comment in ci.yml, the corpus test's own Javadoc and method name, word-fidelity.ps1's -Update help, and FidelityAcceptance's Javadoc.

Verification

  • The export does not change. The 71-document corpus exports byte for byte as before — every document, part, paragraph and report note identical — which is what a test-scope change should do.
  • The Word baseline needs no rewrite, and that is the proof the rule starts green. Word 16.0.20527 over the whole corpus measured all 71 documents and 5165 line rows exactly as word-windows.tsv holds them, not one row differing.
  • DocxFidelityGateTest: 37 tests, from 27. Every rule is pinned in both directions, including a sweep proving a step of exactly half a point is rounding for every drift from 0 to 5pt nearer and further — and that sweep is load-bearing: without the rounding, 36 of those steps each way come out over half a point in doubles.
  • Twenty sabotages, each caught by those tests. Every rule was broken on its own, both by removing it and — where it has a direction — by making it one-sided again: the drift comparison, the editor's page count, the engine's page count and the page's line count each made one-sided; the slack dropped to zero and doubled; the gained-lines, lost-lines, engine-pages, line-count and across rules cut out; the across and down NaN guards reopened; each non-finite refusal removed; note-keeping dropped from both files and the header's place made a guess; the step's difference dropped from the message. Both files byte-identical afterwards.
  • Full reactor clean verify as CI runs it, the four JDK 17 knowledge checks, core javadoc at 0 warnings, and the examples tests: GATE PASS. Note for the reader: clean verify does not run the opt-in corpus test, so it says nothing about the baselines below — CI's DOCX Fidelity (LibreOffice) job is what measures those.

The two LibreOffice baselines, which this change had to look at

Neither has been shot whole since #898 (9f69528b7); since then both were only patched where the one-sided gate went red (#908 53 rows, #971 18, #973 7, #974 46 — and #974, an export change, did not touch the Windows one at all). libreoffice-linux-lines.tsv even carried a note recording ten lines deliberately left 0.1 to 0.3pt further. They are the one-sided era's own sediment, and the two-sided rule reads them differently, so each had to be measured rather than assumed.

libreoffice-linux.tsv is rewritten here, and its first run is the best evidence this change works. This PR's DOCX Fidelity (LibreOffice) run went red on 8 documents and 230 lines. Measured against the old predicate the same run is 0 documents red — it was green before this change — and of its 230 steps past the slack every one stands nearer the engine's PDF and not one further, with no line lost or gained and no page or line count moved, on the build the file names. The baseline was stale purely on improvements the one-sided rule had swallowed. It is re-shot from that run (40844463596): 1778 of 5149 rows had gone stale, 230 of them past the slack — 226 across, 4 down — all nearer. The file's note records the run and that reckoning, and the explanatory cv-teal_pulse note is carried over; three pure-provenance notes are dropped, since a whole re-shoot makes them untrue.

libreoffice-windows.tsv is left as it stands and marked stale in the file itself, because it is the opposite case. Nothing in CI runs it, and it is red already under the rule it was written for: 2 documents, 8 lines no longer found, 2 page counts, and 65 lines more than half a point further. The worst is proposal-indigo, 8 lines 10.25pt lower than the baseline holds them on the LibreOffice build it names (bf6ef2662401), so the export moved them and not the editor. Re-shooting it alongside the Linux file would bury a 10pt move further from the page in a baseline diff, which is what this change exists to stop; a move further wants attributing to the commit that made it first, as the Word baseline's drift was attributed before #975 rewrote it. The file now says so, and so do CONTRIBUTING.md and the CHANGELOG.

One limit worth stating plainly: the slack is read afresh against each rewrite, so it bounds no total — a line may be moved half a point by every change and each step passes as rounding. What holding both ways buys is that every move past rounding is seen and written down; bounding the total is FidelityAcceptance's job.

One correction to this branch's own history

The second commit's message lists the page's line count among the holes it closed. That rule came with the first commit; what the second one added there was the documentation of it. The message is already pushed and rewriting it would mean a force-push, so it stands with that one sentence wrong — this description is the accurate one, and is the better source for a squash message. Its "34 tests from 27" is likewise the PR total at that point, not that commit's step, which was 32 to 34.

… a line moved

The fidelity gate failed a document only when it moved away from the page: a
line more than 0.5pt further than its baseline held it, a line no longer found,
or a page count further from the engine's. Measured on the gate as it stood,
everything else passed in silence — a line 1.10pt nearer the page, one 5pt
nearer, a page count arriving on the engine's own, a line newly found, a line
more drawn on the page, and `+3.00 -> -3.00pt`, which is six points of movement
that a rule comparing which of two drifts is further from the page cannot see at
all, since both stand three points away.

Silence there is not tidiness. A move nearer the page that nothing writes down
leaves the further value in the baseline, so a later change may walk the line
back to it and spend the 0.5pt on top; held that way, the Word baseline went
stale in 1609 of its 5165 rows over 74 commits without one red run, 227 of them
standing as much as a point nearer the page across after #944 and #948, until
three documents stood 0.48pt further and the next change to touch them would
have failed for drift it did not cause.

So each measurement is now held as a step from where the baseline puts it:
`FidelityBaseline.divergences` fails a line whose drift moves more than 0.5pt,
down or across, nearer the page or further; a document the editor sets on another
number of pages, or that the engine draws on another number of pages or lines;
a document whose set of found lines is not the set the baseline holds; and a line
the baseline holds a place across for that is measured without one. Each failure
names the step and its difference, so its direction is in the message, and the
remedy is in the assertion: a change that moves documents, nearer the page
included, rewrites the baseline in the same commit, and that diff is the record.
The slack stays one number for both directions, because it is rounding and
rounding is the same size whichever way a line rounds; the row aggregates are
deliberately held to nothing, since they follow from the lines, which are held
one by one.

The export does not change — the 71-document corpus exports byte for byte as
before — and the Word baseline needs no rewrite: Word 16.0.20527 measures all 71
documents and 5165 line rows exactly as word-windows.tsv holds them, not a row
differing. DocxFidelityGateTest covers each rule in both directions, 32 tests
from 27, including that a step of exactly half a point is rounding for every
drift from 0 to 5pt either way. Thirteen sabotages — every rule removed, and
every rule that has a direction made one-sided again — were each caught by those
tests.

libreoffice-windows.tsv is left as it stands and named as stale instead. Neither
LibreOffice baseline has been shot whole since #898, and the local run is red
already under the one-sided rule: 2 documents, 8 lines no longer found, 2 page
counts and 65 lines more than half a point further, 8 of them proposal-indigo's,
10.25pt lower than it holds them on the LibreOffice build the baseline names.
Rewriting it here would bury a 10pt export move in a baseline diff; it wants
attributing to the commit that moved it first, as the Word baseline's drift was
attributed before #975 rewrote it.
…ate's own NaN hole

The first run of the two-sided gate went red on the Linux LibreOffice baseline:
8 documents, 230 lines. Measured against the old predicate the same run is green,
and of the 230 steps past the slack every one stands nearer the engine's PDF and
not one further, with no line lost or gained and no page or line count moved, on
the build the file names. Not shot whole since #898 and patched since only where
the one-sided gate went red, the baseline was stale on 1778 of its 5149 rows
purely in improvements that rule had swallowed. It is re-shot from that run, and
its note records the run and the reckoning; the note explaining where
cv-teal_pulse's email line stands is carried over, and three notes that only
marked which run a document's rows came from are dropped, since a whole re-shoot
makes them untrue.

libreoffice-windows.tsv is the opposite case and is left alone, marked stale in
the file rather than only in CONTRIBUTING: nothing in CI runs it, and it fails
even one-sidedly on 2 documents — 8 lines no longer found, 2 page counts, 65
lines more than half a point further, 8 of them proposal-indigo's at 10.25pt on
the build it names. A move further from the page wants attributing to the commit
that made it before the file is shot again.

Three holes the gate had, found by reading it against its own thesis:

- A baseline row whose drift or place across was not a number was read, and held
  its line to nothing: every comparison against NaN is false, so a line 500pt
  from where the baseline put it produced no divergence. These files are patched
  by hand, so one mistyped cell was enough. Such a row is now refused, which is
  the mirror of the rule this change added for a measurement carrying no place
  across.
- A rewrite dropped every hand-written note the file carried, and a rewrite is
  now the ordinary answer to a document moving. FidelityBaseline.write keeps them
  under the new header, for the author to prune.
- The page's own line count was not held, so a line added to the page with the
  same ones found passed in silence. It is held beside the pages the engine draws.

The drift figures the javadoc, the changelog and CONTRIBUTING quote were wrong
and are recounted from the committed rows: of the Word baseline's 1609 stale
rows, 1388 moved down the page — 717 nearer it, 635 further, 36 crossing it to
stand the same distance on the other side — and 229 moved across, 228 of them
nearer by as much as 1.10pt. 228 rows had moved more than the slack, 226 across
and 2 down, and no row's distance from the page had grown by more than 0.41pt,
which is why a rule reading distances stayed quiet. CONTRIBUTING also gains the
two engine-side rules it did not list, and both it and the javadoc now say that
the slack is read afresh against each rewrite and so bounds no total.

34 tests from 27, and 17 sabotages — every rule removed, every rule that has a
direction made one-sided again, the non-finite refusals removed and note-keeping
dropped — each caught by DocxFidelityGateTest.
…losed on both axes

The figure the javadoc, the changelog and the pull request quoted for how far
the stale rows had grown — 0.41pt — was the largest growth among the rows whose
step passed the slack, not among the rows that moved, and it contradicted the
next clause of its own sentence. Recounted from the committed rows: no row's
distance from the page grew by more than 0.48pt, 21 rows of cv-serif_headline,
cv-panel and invoice-obsidian by that much and the rest by 0.36pt or less, and
those three documents are the ones that ended 0.48pt further. The argument is
the same and now agrees with itself: 0.48 against half a point of slack is why a
rule reading distances stayed quiet.

Three things the gate still let through:

- A measurement carrying no number for its drift down passed every rule, since
  Math.round(NaN) is 0 and the step against it compares false. The across axis
  had that guard; the down axis did not, so the mirror the last change claimed
  was one-sided. Both axes fail closed now.
- A baseline row's median and p90 were read unchecked, and a median that is not
  a number reads as passing every limit it is put to. They are refused as the
  drift already is.
- Keeping a file's notes did not reach the path the contributor is told to take:
  a measured run writes its own file, which carries no notes, and a run on a
  machine the baseline cannot be shot on is copied over the baseline from there.
  write takes the notes from the baseline, so the copy carries them.

Where a file's header stands is now its shape rather than a guess. The header
cannot be told from a note by its text — both are a # row of prose — so a note
written above it could not be kept without dropping the header in its place and
leaving the file naming a build it was never shot on. notesOf takes the first
line to be the header and refuses a file that does not open with one.

CONTRIBUTING no longer says neither LibreOffice baseline has been shot whole
since #898, which the Linux one now has, and it, the changelog and the job
comment name the engine's two rules in their lists.

37 tests from 27, and 20 sabotages — each rule removed, each rule that has a
direction made one-sided, both NaN guards reopened, every non-finite refusal
removed, note-keeping dropped from either file and the header's place made a
guess — every one caught by DocxFidelityGateTest.
@DemchaAV
DemchaAV marked this pull request as ready for review October 29, 2026 10:12
@DemchaAV
DemchaAV merged commit f29dd10 into 2.5-dev Oct 29, 2026
13 checks passed
@DemchaAV
DemchaAV deleted the fix/docx-fidelity-gate-both-directions branch October 29, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant