Repository navigation
test(qa): rewrite the Word fidelity baseline the export drifted from - #975
Merged
Merged
Conversation
The baseline was shot on Word 16.0.20430, and the Word the gate is run with is 16.0.20527. It sets 1388 of the corpus's 5165 lines at another height and 229 at another place across, 1609 rows in all, reaching 0.48pt further from the page against the 0.51pt at which the gate fails a line: the Word gate passed with 0.03pt to spare on cv-serif_headline, cv-panel and invoice-obsidian. Nine documents already carried rows shot on 16.0.20527, so the file held two builds at once. word-windows.tsv and word-windows-lines.tsv now hold what Word 16.0.20527 measures across all 71 documents, under one header. CONTRIBUTING.md says, where it tells a contributor to run the Word gate, that the baseline's header records the build it was shot on and that a failure on another build is re-shot in a change of its own. Nothing else changes: no Java, and no resource the export reads, so the corpus exports byte for byte as before (71 documents, 1499 report notes). The file holds its own control group: the nine documents whose rows were already shot on 16.0.20527 move 0 lines of 883, while the other 62 move 1609 of 4282. The move is Word's 1/600in grid - 1385 of the 1388 lines whose drift down changes move by an exact multiple of 0.12pt, and 227 of the 229 that move across do too. The other five move a whole number of steps plus 0.02 to 0.03pt. All 5165 lines are found in both measurements, none lost and none new; past 0.05pt, 632 stand nearer the page and 553 further, and across all but one of the 229 move nearer, four invoices' figures by about a point. Of the 47 rows that change, 21 stand nearer, 21 further and 5 the same, no median by more than 0.13pt. Acceptance stays at 63 of 71, the same eight documents short of it. Three full runs - export, a fresh Word conversion of all 71 documents, measure - wrote byte-identical measurements, the baseline is that measurement byte for byte, and the third holds a fresh Word run to the rewritten files.
The note added with the baseline rewrite said a Word build sets lines a little differently on its own, and that a gate failure on another build is re-shot in a change of its own. Measurement says otherwise: the corpus exported from 6683be9, the commit whose run last took all 62 rows on Word 16.0.20430, and converted by Word 16.0.20527 measures byte-identical to that commit's baseline - 62 document rows and 4282 line rows, 0 changes. The two builds read this corpus the same, so the note pointed a contributor at the wrong cause. What actually leaves a baseline stale is the gate's own shape: it fails a line only once it stands more than 0.5pt further from the page, so a change that moves documents nearer, or less than that further, passes and the rows stay as they were. The note now says that, with what it cost between #899 and #974, and keeps the build in one clause as the thing it rarely is.
DemchaAV
changed the title
A cold pass over the note's numbers against the measurements found three wrong: - 75 commits between #899 and #974; `git rev-list --count` says 74, which is also what the per-commit attribution sums to. - "229 of them standing a point nearer the page": 229 lines moved across, but one of them stands further (`proposal-editorial`'s "Signature:", -0.01 -> +0.11pt), and only the 220 of #944 moved by about a point - the other nine moved 0.02 to 0.76pt. - "Word 16.0.20430 and 16.0.20527 measure this corpus identically": the A/B that shows it was run on the corpus as 6683be9 exported it, 62 documents. Today's corpus is 71 and carries least line heights, per-page margin sections and panel cuts that did not exist then, none of which the older build was ever measured on. The sentence now says which corpus it is about.
DemchaAV
marked this pull request as ready for review
October 28, 2026 23:42
DemchaAV
added a commit
that referenced
this pull request
Oct 29, 2026
… a line moved (#976) * test(qa): hold the DOCX fidelity corpus to its baseline whichever way a line moved The fidelity gate failed a document only when it moved away from the page: a line more than 0.5pt further than its baseline held it, a line no longer found, or a page count further from the engine's. Measured on the gate as it stood, everything else passed in silence — a line 1.10pt nearer the page, one 5pt nearer, a page count arriving on the engine's own, a line newly found, a line more drawn on the page, and `+3.00 -> -3.00pt`, which is six points of movement that a rule comparing which of two drifts is further from the page cannot see at all, since both stand three points away. Silence there is not tidiness. A move nearer the page that nothing writes down leaves the further value in the baseline, so a later change may walk the line back to it and spend the 0.5pt on top; held that way, the Word baseline went stale in 1609 of its 5165 rows over 74 commits without one red run, 227 of them standing as much as a point nearer the page across after #944 and #948, until three documents stood 0.48pt further and the next change to touch them would have failed for drift it did not cause. So each measurement is now held as a step from where the baseline puts it: `FidelityBaseline.divergences` fails a line whose drift moves more than 0.5pt, down or across, nearer the page or further; a document the editor sets on another number of pages, or that the engine draws on another number of pages or lines; a document whose set of found lines is not the set the baseline holds; and a line the baseline holds a place across for that is measured without one. Each failure names the step and its difference, so its direction is in the message, and the remedy is in the assertion: a change that moves documents, nearer the page included, rewrites the baseline in the same commit, and that diff is the record. The slack stays one number for both directions, because it is rounding and rounding is the same size whichever way a line rounds; the row aggregates are deliberately held to nothing, since they follow from the lines, which are held one by one. The export does not change — the 71-document corpus exports byte for byte as before — and the Word baseline needs no rewrite: Word 16.0.20527 measures all 71 documents and 5165 line rows exactly as word-windows.tsv holds them, not a row differing. DocxFidelityGateTest covers each rule in both directions, 32 tests from 27, including that a step of exactly half a point is rounding for every drift from 0 to 5pt either way. Thirteen sabotages — every rule removed, and every rule that has a direction made one-sided again — were each caught by those tests. libreoffice-windows.tsv is left as it stands and named as stale instead. Neither LibreOffice baseline has been shot whole since #898, and the local run is red already under the one-sided rule: 2 documents, 8 lines no longer found, 2 page counts and 65 lines more than half a point further, 8 of them proposal-indigo's, 10.25pt lower than it holds them on the LibreOffice build the baseline names. Rewriting it here would bury a 10pt export move in a baseline diff; it wants attributing to the commit that moved it first, as the Word baseline's drift was attributed before #975 rewrote it. * test(qa): shoot the Linux LibreOffice baseline whole, and close the gate's own NaN hole The first run of the two-sided gate went red on the Linux LibreOffice baseline: 8 documents, 230 lines. Measured against the old predicate the same run is green, and of the 230 steps past the slack every one stands nearer the engine's PDF and not one further, with no line lost or gained and no page or line count moved, on the build the file names. Not shot whole since #898 and patched since only where the one-sided gate went red, the baseline was stale on 1778 of its 5149 rows purely in improvements that rule had swallowed. It is re-shot from that run, and its note records the run and the reckoning; the note explaining where cv-teal_pulse's email line stands is carried over, and three notes that only marked which run a document's rows came from are dropped, since a whole re-shoot makes them untrue. libreoffice-windows.tsv is the opposite case and is left alone, marked stale in the file rather than only in CONTRIBUTING: nothing in CI runs it, and it fails even one-sidedly on 2 documents — 8 lines no longer found, 2 page counts, 65 lines more than half a point further, 8 of them proposal-indigo's at 10.25pt on the build it names. A move further from the page wants attributing to the commit that made it before the file is shot again. Three holes the gate had, found by reading it against its own thesis: - A baseline row whose drift or place across was not a number was read, and held its line to nothing: every comparison against NaN is false, so a line 500pt from where the baseline put it produced no divergence. These files are patched by hand, so one mistyped cell was enough. Such a row is now refused, which is the mirror of the rule this change added for a measurement carrying no place across. - A rewrite dropped every hand-written note the file carried, and a rewrite is now the ordinary answer to a document moving. FidelityBaseline.write keeps them under the new header, for the author to prune. - The page's own line count was not held, so a line added to the page with the same ones found passed in silence. It is held beside the pages the engine draws. The drift figures the javadoc, the changelog and CONTRIBUTING quote were wrong and are recounted from the committed rows: of the Word baseline's 1609 stale rows, 1388 moved down the page — 717 nearer it, 635 further, 36 crossing it to stand the same distance on the other side — and 229 moved across, 228 of them nearer by as much as 1.10pt. 228 rows had moved more than the slack, 226 across and 2 down, and no row's distance from the page had grown by more than 0.41pt, which is why a rule reading distances stayed quiet. CONTRIBUTING also gains the two engine-side rules it did not list, and both it and the javadoc now say that the slack is read afresh against each rewrite and so bounds no total. 34 tests from 27, and 17 sabotages — every rule removed, every rule that has a direction made one-sided again, the non-finite refusals removed and note-keeping dropped — each caught by DocxFidelityGateTest. * test(qa): count the drift the gate's own reckoning quotes, and fail closed on both axes The figure the javadoc, the changelog and the pull request quoted for how far the stale rows had grown — 0.41pt — was the largest growth among the rows whose step passed the slack, not among the rows that moved, and it contradicted the next clause of its own sentence. Recounted from the committed rows: no row's distance from the page grew by more than 0.48pt, 21 rows of cv-serif_headline, cv-panel and invoice-obsidian by that much and the rest by 0.36pt or less, and those three documents are the ones that ended 0.48pt further. The argument is the same and now agrees with itself: 0.48 against half a point of slack is why a rule reading distances stayed quiet. Three things the gate still let through: - A measurement carrying no number for its drift down passed every rule, since Math.round(NaN) is 0 and the step against it compares false. The across axis had that guard; the down axis did not, so the mirror the last change claimed was one-sided. Both axes fail closed now. - A baseline row's median and p90 were read unchecked, and a median that is not a number reads as passing every limit it is put to. They are refused as the drift already is. - Keeping a file's notes did not reach the path the contributor is told to take: a measured run writes its own file, which carries no notes, and a run on a machine the baseline cannot be shot on is copied over the baseline from there. write takes the notes from the baseline, so the copy carries them. Where a file's header stands is now its shape rather than a guess. The header cannot be told from a note by its text — both are a # row of prose — so a note written above it could not be kept without dropping the header in its place and leaving the file naming a build it was never shot on. notesOf takes the first line to be the header and refuses a file that does not open with one. CONTRIBUTING no longer says neither LibreOffice baseline has been shot whole since #898, which the Linux one now has, and it, the changelog and the job comment name the engine's two rules in their lists. 37 tests from 27, and 20 sabotages — each rule removed, each rule that has a direction made one-sided, both NaN guards reopened, every non-finite refusal removed, note-keeping dropped from either file and the header's place made a guess — every one caught by DocxFidelityGateTest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Word fidelity gate was passing
2.5-devby two hundredths of a point. 17 lines ofcv-serif_headline,cv-panelandinvoice-obsidianstood 0.48pt further from the page thanword-windows.tsvholds them —cv-serif_headline's certifications at +0.45 → +0.93pt —against the 0.50pt a line may grow by before
FidelityBaseline.regressionscalls it aregression. The next change to touch those documents would have failed the gate for drift it
did not cause.
The baseline's header said Word 16.0.20430 and the Word it is read against here is 16.0.20527,
so the first reading was that the newer build measures the corpus differently. It does not.
The corpus exported from
6683be954— #899, the commit whose run last took all 62 rows on16.0.20430 — and converted by Word 16.0.20527 measures byte-identical to that commit's
committed baseline: 62 document rows and 4282 line rows, 0 changes. On the corpus as #899
exported it, the two builds agree row for row.
What the baseline had fallen behind is the export. The gate fails a line only once it stands
more than 0.50pt further from the page, so a change that moves documents nearer — or less
than that further — passes and leaves the rows as they were. Across the 74 commits from #899 to
#974 that happened 1645 times, and nothing said so.
What changed
word-windows.tsvandword-windows-lines.tsvhold what Word 16.0.20527 measures todayacross all 71 documents, under one header instead of a 16.0.20430 header and three comments
naming nine documents spliced in later.
CONTRIBUTING.md, where it tells a contributor to run the Word gate, now says why a stalebaseline goes unnoticed and what it cost here, and keeps the build as the thing it rarely is.
No Java, and no resource the export reads: the corpus exports byte for byte as
2.5-devdoes —71 documents, 1499 report notes, unchanged — so nothing here can carry an export change of its
own, and nothing it touches reaches a published artifact.
Measured
The build.
Application.Buildover COM:16.0.20527; the conversion records it beside thePDFs. The corpus test's
wordmode checks the SHA-256 the conversion recorded against thistree's own export, so what is measured is Word's PDF of these DOCX. The A/B above is the same
script run on
6683be954, whose sidecar records16.0 (16.0.20527).The engine did not move.
driftis the editor's line top less the engine's. The engine's ownPDFs are byte-identical for all 62 documents at every commit measured below (compared with the
time-seeded trailer stripped), so the whole move is in the DOCX. 59 of those 62 DOCX changed over
the window; the three that did not —
cv-modern_professional,letter-editorial_blue,letter-modern_professional— move 0 of their 83 lines.Every move is attributed to the change that made it. Thirteen Word measurements across the
window, each a full export, conversion and measurement, plus corpus exports at seven more
commits to narrow them:
cv-terracotta_rail, −0.12pt eachThe commit counts sum to the 74 in
6683be954..ffbc08f0e. The 1645 moves land on 1614 lines —1393 down, 229 across, 8 on both, 23 moving twice down and 5 ending where they started — which
leaves the 1609 rows that differ end to end. Across, 220 + 3 + 6 = 229 exactly.
Across, a point nearer the page. #944 (
2defe85e7, an auto column's editor slack held inits cells' margins) brought four invoices' and a proposal's right-aligned figures from about a
point off to within 0.13pt:
invoice-classic_long92 lines, "GBP 698" +1.05 → +0.09pt;invoice-modern_long92;invoice-classic17, "Total" +0.98 → +0.02pt;invoice-modern17;proposal-modern2, +1.12 → +0.02pt. #948 (5c2e62b6d) movedproposal-editorial's "Date:"and "Signature:" and one
cv-violet_gridline; #949 (8400edd40, a panel set in from itscell's text set where the page sets its text) moved
cv-charcoal_gold's six sidebar headings0.84pt across, to within 0.12pt of the page. 228 of the 229 end nearer the page, 227 of them by
more than 0.05pt; the one that ends further is
proposal-editorial's "Signature:",−0.01 → +0.11pt.
Down, the declared cost of growing lines with their text. #955 to #959 wrote a line as a
least height where it is at least as tall as its faces' own, so Word cannot clip text a reader
enlarges. #955 alone moved 1082 lines in 51 documents, and it is the commit that put the gate
two hundredths from failing — the same 17 lines,
cv-serif_headline13 (+0.45 → +0.93,+0.49 → +0.97, +0.56 → +1.04),
cv-panel2 (+0.10 → +0.58),invoice-obsidian2(+0.17 → +0.65), with five more
cv-panellines past 0.40pt. Its steps: −0.48 ×3, −0.36 ×6,−0.24 ×52, −0.12 ×148, +0.12 ×331, +0.24 ×387, +0.36 ×84, +0.48 ×67, and one each of +0.14,
+0.26, +0.51 and +0.60. Its own report measured the corpus's restyle failures down from 1617
paragraphs to 1138 and the documents passing up from 2 to 10, which is what it was for.
None of it is a regression. Every line that ends further moved because a change said it
would move lines, in the direction that change describes and by one to four steps of Word's
grid.
FidelityBaseline.furtherfires on no line at any of the thirteen measurements or end toend; no line was lost or gained; no document's page count, line count or lines-found changes.
Nothing anywhere in the window ends more than 0.48pt further from the page — though a
cv-panelline is displaced 0.60pt at #955, and #944's figures 1.10pt across, both towards thepage.
The grid. Word sets a line's top on a 1/600in grid, 0.12pt. 1385 of the 1388 lines whose
drift down changes end to end move by an exact multiple of it, and 227 of the 229 across. The
five that do not move a whole number of steps plus 0.02 to 0.03pt:
cv-panel's "TemplateStudio" list item,
cv-serif_headline's LinkedIn line,cv-violet_grid's "UX / UI DESIGNER"heading, and
proposal-modern's two −1.10pt figures.What the rewrite holds. 5165 lines are found in both measurements, none lost and none new,
and no page count, line count or lines-found changes. Down, past 0.05pt, 632 stand nearer the
page and 553 further. 47 document rows change: 21 nearer, 21 further, 5 the same, no median
moving more than 0.13pt; three more documents keep their row and still hold a line past 0.05pt
further (
cv-mint_editorial,letter-engineering_resume,letter-panel).invoice-meteredcv-violet_gridcv-terracotta_railcv-panelinvoice-paymentsletter-blue_bannercv-serif_headlineAcceptance is 63 of 71 before and after, the same eight documents short of it: six on the
share of lines found,
synthetic-icon_over_cardon both its median of 13.99pt and three linespast 2pt,
synthetic-card_over_pageson a median of 1.03pt.Reproducibility. Three full runs — export, a fresh Word conversion of all 71 documents,
measure, each from a deleted output folder — wrote byte-identical
measured-word.tsvandmeasured-word-lines.tsv, and the committed baseline is that measurement byte for byte.-Updateprinted nothing as rewritten over: 0 lines in the new measurement would fail the oldbaseline, so the rewrite suppresses no gate failure.
Named, not fixed
it, so a change that improves a document by a point leaves the baseline stale and nothing says
so. That is how fix(docx): hold an auto column's editor slack in its cells' margins #944's 220 lines went unrecorded and how fix(docx): grow lines with their text where they are as tall as their faces' own #955's 0.48pt reached two hundredths
of the allowance.
percentile improving from 0.281 to 0.265pt, while its worst line went from +0.45 to +0.93pt.
It named the equivalent tail for LibreOffice — "the largest move away is 0.50pt … the gate's
slack is 0.5" — and no figure for Word. A report should name the worst line for the editor it
gates on.
invoice-metered_longstands 0.50ptor more off across on 278 of its 350 lines, up to 1.01pt;
invoice-obsidian_longon 140 of375, up to 0.77pt;
invoice-consulting_longon 49 of 236, up to 0.96pt. The classic andmodern long invoices no longer do.
FidelityBaseline.readdrops every#line, so the build abaseline records is prose; the run's build is already in hand in
DocxFidelityCorpusTest.Verification
word-fidelity.ps1against the committed baseline, before the rewrite → BUILD SUCCESS,2 tests: the old baseline still held, by two hundredths of a point.
word-fidelity.ps1 -Update→ BUILD SUCCESS, 2 tests, nothing printed as rewritten over.word-fidelity.ps1again, against the rewritten baseline → BUILD SUCCESS: the gate readsthe new files and holds a fresh Word run to them. A reactor
clean verifycannot show this —without
-Dgraphcompose.docxFidelitythe corpus test is skipped, and CI runs it withLibreOffice.
word-fidelity.ps1on6683be954with this machine's Word → BUILD SUCCESS, and itsmeasurement equals that commit's 16.0.20430 baseline row for row. That is the A/B.
6683be954..ffbc08f0efor the attribution table, andcorpus exports at seven more commits to narrow it.
1499 → 1499.
clean verifyof the reactor as CI runs it, on this branch's head90f254a54→BUILD SUCCESS: core 823, render-pdf 339, render-docx 2334, testing 5, render-pptx 142,
templates 127, qa 1846. The four JDK 17 knowledge checks pass, core javadoc has 0 warnings,
and
examplespasses 93 tests,CommittedAssetDriftTestincluded. The core documentationguards that read
CONTRIBUTING.mdare in that run, on the wording this branch ships.Lane: test — a qa fidelity baseline and the contributor note for it. No public API, no
export change, no CHANGELOG entry: the baseline rewrites before this one carried none either.