Skip to content

Add experimental native English normalization runtime - #512

Open
albertz wants to merge 6 commits into
NVIDIA:mainfrom
albertz:native-english-normalization
Open

albertz wants to merge 6 commits into
NVIDIA:mainfrom
albertz:native-english-normalization

Conversation

@albertz

@albertz albertz commented Oct 17, 2026 •

Copy link
Copy Markdown

Large English text-preparation jobs spend most of their CPU time constructing and evaluating per-input FSTs.
This adds an opt-in C++17 runtime that exports existing Pynini grammars once and executes normalization without an OpenFst runtime dependency or Python callbacks between stages.

The complete English path includes whitespace and punctuation preprocessing, weighted classification, tagged-field permutations, verbalization, postprocessing and detokenization.
It exposes scalar and framed batch APIs plus a standalone stream executable.
Existing Normalizer and Sparrowhawk behavior is unchanged.

Compatibility is deliberately scoped to deterministic, cased English with punctuation processing enabled and the pinned prepared-lookahead classifier semantics.
Ordinary Pynini composition can choose different equally weighted outputs.
This is an experimental backend, not a drop-in replacement for every Normalizer option, grammar or dependency version.
Unsupported labels, invalid UTF-8 and resource limits fail explicitly.
The native batch API identifies the failing record, publishes no partial result and remains usable afterward.

The English pipeline now accepts embedded NUL, matching the reference's input-epsilon behavior instead of rejecting the record.
It preserves the original text for punctuation handling and fallback, while retaining input-epsilon metadata needed for OpenFst queue ordering and terminal lookahead phases.
Simply deleting NUL before normalization does not preserve these semantics.
The low-level transducer's direct-composition mode still rejects NUL explicitly; NUL support is scoped to lookahead and output-projected modes used by the English pipeline.

Validation:

  • The original 20 native regression tests pass on ARM64 and x86-64, covering weighted ties, lookahead phases, Unicode, SIMD/scalar parity, field ordering, malformed batches and retry behavior.
  • All 22 current native tests have passed on x86-64, including NUL at SIMD boundaries, multibyte text, scalar/batch parity, fallback, and fixed regressions for queue ordering and terminal lookahead phases.
  • 22,000 randomized prepared-lookahead queries with initial, interior, trailing and repeated NUL have zero differences against OpenFst, including weighted/unweighted and cached-acyclicity cases.
  • 322 complete English normalization cases, including five real affected corpus chunks and synthetic NUL placements, match the cached-lookahead Python/OpenFst reference exactly.
  • A scheduled 110,000-record preparation comparison produced 132 byte-identical artifacts, including 108,638 normalized records, summaries and LM splits.
    Rechecking those 108,638 normalized records with the NUL-compatible implementation gives zero differences.
  • An additional 2,200-record x86-64 comparison has zero differences through scalar and batch APIs.
    The same batch passes AddressSanitizer and UndefinedBehaviorSanitizer.
  • The earlier ARM64 benchmark on one CPU measured 29.43 seconds versus 644.23 seconds for matched normalization calls (21.9×).
    Including decoding and output processing, the loop took 38.32 versus 670.91 seconds (17.5×).
    These measurements precede the NUL correction; repeated environment extraction reduces the overall small-job gain, and the figures are not universal throughput guarantees.
  • The repository CPU pytest entry point passed all 1,635 English and native tests before the NUL correction (566.54 seconds).
    The full multilingual suite and Sparrowhawk Docker suite have not been run here.

Ready for review of the runtime approach and compatibility boundary.
In particular, queue/tie semantics and the supported options need upstream agreement before treating this as a supported backend.
Finite differential tests do not prove equivalence for every possible input.

albertz and others added 2 commits October 17, 2026 06:02
@albertz
albertz marked this pull request as ready for review October 17, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant