Repository navigation
Conversation
Signed-off-by: Albert Zeyer <azeyer@nvidia.com>
for more information, see https://pre-commit.ci
albertz
marked this pull request as ready for review
October 17, 2026 17:01
Signed-off-by: Albert Zeyer <azeyer@nvidia.com>
for more information, see https://pre-commit.ci
Signed-off-by: Albert Zeyer <azeyer@nvidia.com>
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large English text-preparation jobs spend most of their CPU time constructing and evaluating per-input FSTs.
This adds an opt-in C++17 runtime that exports existing Pynini grammars once and executes normalization without an OpenFst runtime dependency or Python callbacks between stages.
The complete English path includes whitespace and punctuation preprocessing, weighted classification, tagged-field permutations, verbalization, postprocessing and detokenization.
It exposes scalar and framed batch APIs plus a standalone stream executable.
Existing Normalizer and Sparrowhawk behavior is unchanged.
Compatibility is deliberately scoped to deterministic, cased English with punctuation processing enabled and the pinned prepared-lookahead classifier semantics.
Ordinary Pynini composition can choose different equally weighted outputs.
This is an experimental backend, not a drop-in replacement for every Normalizer option, grammar or dependency version.
Unsupported labels, invalid UTF-8 and resource limits fail explicitly.
The native batch API identifies the failing record, publishes no partial result and remains usable afterward.
The English pipeline now accepts embedded NUL, matching the reference's input-epsilon behavior instead of rejecting the record.
It preserves the original text for punctuation handling and fallback, while retaining input-epsilon metadata needed for OpenFst queue ordering and terminal lookahead phases.
Simply deleting NUL before normalization does not preserve these semantics.
The low-level transducer's direct-composition mode still rejects NUL explicitly; NUL support is scoped to lookahead and output-projected modes used by the English pipeline.
Validation:
Rechecking those 108,638 normalized records with the NUL-compatible implementation gives zero differences.
The same batch passes AddressSanitizer and UndefinedBehaviorSanitizer.
Including decoding and output processing, the loop took 38.32 versus 670.91 seconds (17.5×).
These measurements precede the NUL correction; repeated environment extraction reduces the overall small-job gain, and the figures are not universal throughput guarantees.
The full multilingual suite and Sparrowhawk Docker suite have not been run here.
Ready for review of the runtime approach and compatibility boundary.
In particular, queue/tie semantics and the supported options need upstream agreement before treating this as a supported backend.
Finite differential tests do not prove equivalence for every possible input.