Repository navigation
Replace fasttext with lingua and overhaul wrong-language detection - #1
Merged
Merged
Conversation
fasttext-wheel has no wheels past cp312 and pins numpy<2, which blocked installing on Python 3.13+ (dashboard#2091). lingua 2.x is a Rust engine with cp312-cp314 wheels and zero dependencies, and its models ship in the wheel so nothing is downloaded at runtime. The detection pipeline (carrier phrases, confused merges, thresholds) carries over unchanged apart from two new directional Cyrillic merges lingua needs: Ukrainian text with dotted i reads as Kazakh, and short Russian phrases read as Bulgarian. Bokmal and Nynorsk confidences are folded into 'no' since lingua has no Norwegian macrolanguage, and locales lingua has no model for are skipped instead of flagged. --compact-model now maps to lingua's low accuracy mode. Requires Python 3.12+ (lingua's wheel floor). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion
The carrier phrases, confused-merge tables, and 0.5 absolute threshold were
all tuned to fasttext's top-k output. Benchmarked against the full PESCHECK
translation corpus (27k entries) plus 6800 planted contaminations, they were
counterproductive on lingua: the merge tables made whole sibling pairs
undetectable (0/25 planted recall on pt<-es and da<-sv) while still letting
mixed-script brand strings flag as exotic languages.
Replaced with three mechanisms measured to give zero false positives on the
corpus at 88.7% planted recall:
- The detector is restricted to the locales present in the linted dirs plus
the source language, instead of all 70+ spoken languages. Contamination
comes from neighbouring rows of the same translation batch, and the
restriction keeps confidence mass on languages that can actually occur.
- A relative rule replaces the absolute threshold: flag only when the
expected language scores below expected_confidence_max (0.05) while
another language tops confidence_threshold (0.7). This asks 'is this
clearly NOT the expected language' instead of 'what is the argmax'.
- Tokens in a script the locale does not use are stripped before detection.
Lingua only considers the dominant script's words, so quoted product
nouns ('Скопіюйте Application ID та Client Secret') otherwise drown out
the native text entirely. All-foreign-script contamination stays caught
by check_wrong_script, which runs first.
Carrier phrases and confused merges are deleted. New expected_confidence_max
config knob and --expected-confidence-max CLI flag; confidence_threshold
default moves 0.5 -> 0.7. Full-corpus lint runs in ~16s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A loan word, product noun, or quoted term in a translation is a token carried over verbatim from the source string, so the msgid (already in the is_wrong_language signature, previously unused) is the exact list of words that say nothing about the translation's language. Dropping msgstr tokens whose words all appear in the msgid isolates the actually translated words; a string that is mostly untranslatable jargon falls under the minimum detection length and is skipped. Benchmarked against the corpus plus synthetic stress suites (English tech tokens injected into correct translations at 20-80% of tokens, and into planted contamination at 30%): zero false positives at every injection ratio, including the quoted-Dutch-law-name lines that previously needed a .po-lint-ignore entry, which is now redundant. With the filter in place expected_confidence_max goes back to 0.1, restoring sibling-pair recall (planted 89.2%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Test matrix 3.12-3.14 to match requires-python (lingua's wheel floor), regenerated uv.lock, and dropped PO_LINT_COMPACT_MODEL from the test run: it exists to skip the old 126MB fasttext download, but lingua's models ship in the wheel, and the thresholds are tuned for high accuracy mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restores installability on Python 3.13/3.14 (pescheck/dashboard#2091 on GitLab): fasttext-wheel tops out at cp312 wheels and pins numpy<2. lingua 2.2 ships cp312-cp314 wheels, has zero dependencies, and bundles its models in the wheel, so nothing is downloaded at runtime.
The fasttext-era tuning (carrier phrases, confused-merge tables, 0.5 absolute threshold) measured counterproductive on lingua, so wrong-language detection is rebuilt around lingua's full confidence distribution:
Verification
Benchmarked against the full PESCHECK translation corpus (27,011 entries, 507 .po files) plus 6,800 planted contaminations (real msgstrs judged as another locale) and synthetic stress suites:
Both real contaminations in the corpus are found: Swedish in examplecheck's sw locale, Spanish in idcheck's it locale (the latter at expected_confidence_max 0.05). Requires Python 3.12+.