Repository navigation
test(qa): hold the DOCX fidelity corpus to its baseline whichever way a line moved - #976
Merged
Merged
Conversation
… a line moved The fidelity gate failed a document only when it moved away from the page: a line more than 0.5pt further than its baseline held it, a line no longer found, or a page count further from the engine's. Measured on the gate as it stood, everything else passed in silence — a line 1.10pt nearer the page, one 5pt nearer, a page count arriving on the engine's own, a line newly found, a line more drawn on the page, and `+3.00 -> -3.00pt`, which is six points of movement that a rule comparing which of two drifts is further from the page cannot see at all, since both stand three points away. Silence there is not tidiness. A move nearer the page that nothing writes down leaves the further value in the baseline, so a later change may walk the line back to it and spend the 0.5pt on top; held that way, the Word baseline went stale in 1609 of its 5165 rows over 74 commits without one red run, 227 of them standing as much as a point nearer the page across after #944 and #948, until three documents stood 0.48pt further and the next change to touch them would have failed for drift it did not cause. So each measurement is now held as a step from where the baseline puts it: `FidelityBaseline.divergences` fails a line whose drift moves more than 0.5pt, down or across, nearer the page or further; a document the editor sets on another number of pages, or that the engine draws on another number of pages or lines; a document whose set of found lines is not the set the baseline holds; and a line the baseline holds a place across for that is measured without one. Each failure names the step and its difference, so its direction is in the message, and the remedy is in the assertion: a change that moves documents, nearer the page included, rewrites the baseline in the same commit, and that diff is the record. The slack stays one number for both directions, because it is rounding and rounding is the same size whichever way a line rounds; the row aggregates are deliberately held to nothing, since they follow from the lines, which are held one by one. The export does not change — the 71-document corpus exports byte for byte as before — and the Word baseline needs no rewrite: Word 16.0.20527 measures all 71 documents and 5165 line rows exactly as word-windows.tsv holds them, not a row differing. DocxFidelityGateTest covers each rule in both directions, 32 tests from 27, including that a step of exactly half a point is rounding for every drift from 0 to 5pt either way. Thirteen sabotages — every rule removed, and every rule that has a direction made one-sided again — were each caught by those tests. libreoffice-windows.tsv is left as it stands and named as stale instead. Neither LibreOffice baseline has been shot whole since #898, and the local run is red already under the one-sided rule: 2 documents, 8 lines no longer found, 2 page counts and 65 lines more than half a point further, 8 of them proposal-indigo's, 10.25pt lower than it holds them on the LibreOffice build the baseline names. Rewriting it here would bury a 10pt export move in a baseline diff; it wants attributing to the commit that moved it first, as the Word baseline's drift was attributed before #975 rewrote it.
…ate's own NaN hole The first run of the two-sided gate went red on the Linux LibreOffice baseline: 8 documents, 230 lines. Measured against the old predicate the same run is green, and of the 230 steps past the slack every one stands nearer the engine's PDF and not one further, with no line lost or gained and no page or line count moved, on the build the file names. Not shot whole since #898 and patched since only where the one-sided gate went red, the baseline was stale on 1778 of its 5149 rows purely in improvements that rule had swallowed. It is re-shot from that run, and its note records the run and the reckoning; the note explaining where cv-teal_pulse's email line stands is carried over, and three notes that only marked which run a document's rows came from are dropped, since a whole re-shoot makes them untrue. libreoffice-windows.tsv is the opposite case and is left alone, marked stale in the file rather than only in CONTRIBUTING: nothing in CI runs it, and it fails even one-sidedly on 2 documents — 8 lines no longer found, 2 page counts, 65 lines more than half a point further, 8 of them proposal-indigo's at 10.25pt on the build it names. A move further from the page wants attributing to the commit that made it before the file is shot again. Three holes the gate had, found by reading it against its own thesis: - A baseline row whose drift or place across was not a number was read, and held its line to nothing: every comparison against NaN is false, so a line 500pt from where the baseline put it produced no divergence. These files are patched by hand, so one mistyped cell was enough. Such a row is now refused, which is the mirror of the rule this change added for a measurement carrying no place across. - A rewrite dropped every hand-written note the file carried, and a rewrite is now the ordinary answer to a document moving. FidelityBaseline.write keeps them under the new header, for the author to prune. - The page's own line count was not held, so a line added to the page with the same ones found passed in silence. It is held beside the pages the engine draws. The drift figures the javadoc, the changelog and CONTRIBUTING quote were wrong and are recounted from the committed rows: of the Word baseline's 1609 stale rows, 1388 moved down the page — 717 nearer it, 635 further, 36 crossing it to stand the same distance on the other side — and 229 moved across, 228 of them nearer by as much as 1.10pt. 228 rows had moved more than the slack, 226 across and 2 down, and no row's distance from the page had grown by more than 0.41pt, which is why a rule reading distances stayed quiet. CONTRIBUTING also gains the two engine-side rules it did not list, and both it and the javadoc now say that the slack is read afresh against each rewrite and so bounds no total. 34 tests from 27, and 17 sabotages — every rule removed, every rule that has a direction made one-sided again, the non-finite refusals removed and note-keeping dropped — each caught by DocxFidelityGateTest.
…losed on both axes The figure the javadoc, the changelog and the pull request quoted for how far the stale rows had grown — 0.41pt — was the largest growth among the rows whose step passed the slack, not among the rows that moved, and it contradicted the next clause of its own sentence. Recounted from the committed rows: no row's distance from the page grew by more than 0.48pt, 21 rows of cv-serif_headline, cv-panel and invoice-obsidian by that much and the rest by 0.36pt or less, and those three documents are the ones that ended 0.48pt further. The argument is the same and now agrees with itself: 0.48 against half a point of slack is why a rule reading distances stayed quiet. Three things the gate still let through: - A measurement carrying no number for its drift down passed every rule, since Math.round(NaN) is 0 and the step against it compares false. The across axis had that guard; the down axis did not, so the mirror the last change claimed was one-sided. Both axes fail closed now. - A baseline row's median and p90 were read unchecked, and a median that is not a number reads as passing every limit it is put to. They are refused as the drift already is. - Keeping a file's notes did not reach the path the contributor is told to take: a measured run writes its own file, which carries no notes, and a run on a machine the baseline cannot be shot on is copied over the baseline from there. write takes the notes from the baseline, so the copy carries them. Where a file's header stands is now its shape rather than a guess. The header cannot be told from a note by its text — both are a # row of prose — so a note written above it could not be kept without dropping the header in its place and leaving the file naming a build it was never shot on. notesOf takes the first line to be the header and refuses a file that does not open with one. CONTRIBUTING no longer says neither LibreOffice baseline has been shot whole since #898, which the Linux one now has, and it, the changelog and the job comment name the engine's two rules in their lists. 37 tests from 27, and 20 sabotages — each rule removed, each rule that has a direction made one-sided, both NaN guards reopened, every non-finite refusal removed, note-keeping dropped from either file and the header's place made a guess — every one caught by DocxFidelityGateTest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The DOCX fidelity gate only ever looked for documents moving away from the page.
FidelityBaseline.regressionsfailed a line that stood more than 0.5pt further from the engine's PDF than its baseline held it, a line the baseline found and the editor no longer sets at those words, and a page count further from the engine's. Measured on the gate as it stood, by feeding it baselines and measurements that differ in one way each:+1.00 -> +1.60pt(further, past the slack)+1.00 -> +0.49pt(0.51pt nearer)+1.10 -> +0.00pt(1.10pt nearer — what #944 and #948 did to 227 lines)+5.00 -> +0.00pt(five points nearer)+3.00 -> -3.00pt(six points of movement, same distance)The fifth row is the one that shows the rule was wrong rather than merely narrow: written as "is the new drift further from the page than the old one", it cannot see a line that crosses the page's own line at the same distance, however far it travels —
+3.00 -> -3.00ptis six points of movement and both values stand three points away.And silence about a move nearer the page is not tidiness. Unwritten, the baseline keeps the further value, so a later change may walk the line back to it and spend the 0.5pt of slack on top of that.
Held one-sidedly, the Word baseline went stale in 1609 of its 5165 rows across 74 commits without a single red run: 1388 rows moved down the page — 717 nearer it, 635 further, and 36 crossing it to stand the same distance on the other side — and 229 moved across, 228 of them nearer by as much as 1.10pt. 228 of those rows had moved more than the slack (226 across and 2 down, both of the latter crossing the page's line). And yet no row's distance from the page had grown by more than 0.48pt — 21 rows of
cv-serif_headline,cv-panelandinvoice-obsidianby that much, the rest by 0.36pt or less — which against half a point of slack is exactly why a rule reading distances stayed quiet. Those same three documents are the ones that ended 0.48pt further, where the next change to touch them would have failed the gate for drift it did not cause.What changed
FidelityBaseline.regressionsbecomesdivergences, and holds every measurement as a step from where the baseline puts it rather than as a distance to compare:LINE_SLACK(0.5pt), down or across, nearer the page or further —Math.abs(now - was), notMath.abs(now)againstMath.abs(was);NaNis false —Math.round(NaN)is 0 — so such a row held its line to nothing however far it moved: a line 500pt out of place produced no divergence at all. These files are patched by hand, so one mistyped cell was enough;-Dupdate=truerun and the CI artifact that gets copied over the Linux baseline. Those notes are where a row's reckoning is written down, and a rewrite is now the ordinary answer to a document moving, so dropping them would spend the record this gate exists to keep. The header is known by standing first, since it cannot be told from a note by its text, and a baseline that does not open with it is refused rather than half-read.Each failure names the step and its difference,
"aa" +1.10 -> +0.00pt (-1.10), so the direction is in the message rather than left for the reader to work out. The remedy is in the assertion itself: a change that moves documents — nearer the page included — rewrites the baseline in the same commit with-Dgraphcompose.docxFidelity.update=true, and that diff is the record of what moved and which way.Two things deliberately left alone:
Documentation that stated the old rule is corrected: two paragraphs of
CONTRIBUTING.md, theDOCX Fidelityjob comment inci.yml, the corpus test's own Javadoc and method name,word-fidelity.ps1's-Updatehelp, andFidelityAcceptance's Javadoc.Verification
word-windows.tsvholds them, not one row differing.DocxFidelityGateTest: 37 tests, from 27. Every rule is pinned in both directions, including a sweep proving a step of exactly half a point is rounding for every drift from 0 to 5pt nearer and further — and that sweep is load-bearing: without the rounding, 36 of those steps each way come out over half a point in doubles.clean verifyas CI runs it, the four JDK 17 knowledge checks, core javadoc at 0 warnings, and the examples tests:GATE PASS. Note for the reader:clean verifydoes not run the opt-in corpus test, so it says nothing about the baselines below — CI'sDOCX Fidelity (LibreOffice)job is what measures those.The two LibreOffice baselines, which this change had to look at
Neither has been shot whole since #898 (
9f69528b7); since then both were only patched where the one-sided gate went red (#908 53 rows, #971 18, #973 7, #974 46 — and #974, an export change, did not touch the Windows one at all).libreoffice-linux-lines.tsveven carried a note recording ten lines deliberately left 0.1 to 0.3pt further. They are the one-sided era's own sediment, and the two-sided rule reads them differently, so each had to be measured rather than assumed.libreoffice-linux.tsvis rewritten here, and its first run is the best evidence this change works. This PR'sDOCX Fidelity (LibreOffice)run went red on 8 documents and 230 lines. Measured against the old predicate the same run is 0 documents red — it was green before this change — and of its 230 steps past the slack every one stands nearer the engine's PDF and not one further, with no line lost or gained and no page or line count moved, on the build the file names. The baseline was stale purely on improvements the one-sided rule had swallowed. It is re-shot from that run (40844463596): 1778 of 5149 rows had gone stale, 230 of them past the slack — 226 across, 4 down — all nearer. The file's note records the run and that reckoning, and the explanatorycv-teal_pulsenote is carried over; three pure-provenance notes are dropped, since a whole re-shoot makes them untrue.libreoffice-windows.tsvis left as it stands and marked stale in the file itself, because it is the opposite case. Nothing in CI runs it, and it is red already under the rule it was written for: 2 documents, 8 lines no longer found, 2 page counts, and 65 lines more than half a point further. The worst isproposal-indigo, 8 lines 10.25pt lower than the baseline holds them on the LibreOffice build it names (bf6ef2662401), so the export moved them and not the editor. Re-shooting it alongside the Linux file would bury a 10pt move further from the page in a baseline diff, which is what this change exists to stop; a move further wants attributing to the commit that made it first, as the Word baseline's drift was attributed before #975 rewrote it. The file now says so, and so doCONTRIBUTING.mdand the CHANGELOG.One limit worth stating plainly: the slack is read afresh against each rewrite, so it bounds no total — a line may be moved half a point by every change and each step passes as rounding. What holding both ways buys is that every move past rounding is seen and written down; bounding the total is
FidelityAcceptance's job.One correction to this branch's own history
The second commit's message lists the page's line count among the holes it closed. That rule came with the first commit; what the second one added there was the documentation of it. The message is already pushed and rewriting it would mean a force-push, so it stands with that one sentence wrong — this description is the accurate one, and is the better source for a squash message. Its "34 tests from 27" is likewise the PR total at that point, not that commit's step, which was 32 to 34.