PII Pattern Engine

A comprehensive collection of regex patterns, verification functions, and a re-rank scoring system for detecting personally identifiable information (PII) and sensitive data across multiple languages and regions.

Overview

PII Pattern Engine provides:

• Regex Patterns: 204 patterns for PII detection across US, Korea, Japan, China, Taiwan, India, EU, Spain, France, and common international formats
• Re-Rank Scoring System: Confidence scoring engine that combines keyword signals, field-label context, and regex evidence into a single 0–100 score
• Reference Registry: Stable numeric codes (1001–9510) for all 153 PII entity types — safe for use in logs, configs, and APIs
• Verification Functions: 55 checksum/validation functions, implemented identically in Python, Go, Java, and JavaScript (behavioral parity is verified against the shared pattern examples)
• Keyword Mappings: Context-aware keywords in multiple languages for improved accuracy
• Comprehensive Tests: 3,500+ automated tests ensuring accuracy and reliability

---

---

Project Structure

pii-pattern-engine/
├── regex/                        # Regex pattern definitions (YAML)
│   ├── pii/                     # Personally Identifiable Information
│   │   ├── us/                 # United States (SSN, passport, phone, etc.)
│   │   ├── kr/                 # South Korea (RRN, bank accounts, phone, etc.)
│   │   ├── jp/                 # Japan (My Number, bank accounts, phone, etc.)
│   │   ├── cn/                 # China (ID cards, bank accounts, phone, etc.)
│   │   ├── tw/                 # Taiwan (ID cards, bank accounts, phone, etc.)
│   │   ├── in/                 # India (Aadhaar, PAN, GST, phone, etc.)
│   │   ├── eu/                 # European Union (VAT, national IDs, passports, etc.)
│   │   ├── es/                 # Spain (DNI, NIE, CIF, phone)
│   │   ├── fr/                 # France (NIR, CNI, passport, phone)
│   │   ├── common/             # International (email, IP, credit cards, URLs, etc.)
│   │   └── iban.yml            # IBAN patterns (DE, FR, GB, IT + generic)
│   ├── hash/                    # High-entropy tokens and secrets
│   │   └── tokens.yml          # API keys, JWT, AWS keys, GitHub tokens, etc.
│   ├── sox/                     # SOX financial control patterns
│   └── tech/                    # Technical identifiers (patents, UUIDs, etc.)
│
├── re-rank/                      # Confidence scoring engine (v2.0)
│   ├── scoring.yml              # Engine config: formula, thresholds, entity index
│   ├── kr.yml                   # Korea entities (codes 1001–1013)
│   ├── jp.yml                   # Japan entities (codes 2001–2016)
│   ├── cn.yml                   # China entities (codes 3001–3012)
│   ├── tw.yml                   # Taiwan entities (codes 4001–4016)
│   ├── us.yml                   # United States entities (codes 5001–5009)
│   ├── eu.yml                   # European Union entities (codes 6001–6054)
│   ├── in.yml                   # India entities (codes 7001–7009)
│   ├── es_fr.yml                # Spain & France entities (codes 8001–8056)
│   ├── common.yml               # Common/international entities (codes 9001–9013)
│   └── tech.yml                 # Tech tokens & secrets (codes 9501–9510)
│
├── reference/                    # Stable numeric code registry
│   ├── registry.yml             # Master flat code map (all 153 entities)
│   ├── kr.yml                   # Korea entity reference
│   ├── jp.yml                   # Japan entity reference
│   ├── cn.yml                   # China entity reference
│   ├── tw.yml                   # Taiwan entity reference
│   ├── us.yml                   # United States entity reference
│   ├── eu.yml                   # European Union entity reference
│   ├── in.yml                   # India entity reference
│   ├── es_fr.yml                # Spain & France entity reference
│   ├── common.yml               # Common/international entity reference
│   └── tech.yml                 # Tech tokens entity reference
│
├── verification/                 # Validation functions (multi-language)
│   ├── python/
│   │   ├── verification.py      # 55 verification functions
│   │   └── __init__.py
│   ├── golang/
│   │   ├── verification.go
│   │   └── go.mod
│   ├── java/
│   │   └── src/main/java/com/piipatternengine/verification/
│   │       └── Verification.java
│   ├── javascript/
│   │   └── verification.js
│   ├── __init__.py
│   ├── README.md
│   └── USAGE.md
│
├── keyword/                      # Context-aware keywords (multi-language)
│   ├── financial.yml
│   ├── identification.yml
│   ├── contact.yml
│   ├── network.yml
│   ├── personal.yml
│   └── README.md
│
├── prefilter/                    # Scan-optimization layer (pre-filter)
│   ├── prefilter.py             # Aho-Corasick keywords + min_digit_run gating
│   └── README.md
│
├── examples/                     # Runnable, end-to-end examples
│   ├── detect.py                # Detection engine using the Aho-Corasick pre-filter
│   └── README.md
│
├── tests/                        # Test suite
│   ├── test_rerank.py           # Re-rank scoring tests
│   ├── test_verification.py     # Verification function tests
│   ├── test_patterns.py         # Pattern validation tests
│   ├── test_prefilter.py        # Pre-filter + hit_keywords/min_digit_run tests
│   ├── test_examples.py         # End-to-end example + pre-filter equivalence tests
│   └── README.md
│
├── INTEGRATION.md                # How to build a detector on this library
├── pytest.ini
└── README.md

Quick Start

Installation

git clone https://github.com/zafrem/pii-pattern-engine.git
cd pii-pattern-engine

# Install as an editable library (pulls in the runtime dependency, PyYAML)
pip install -e .

# Optional extras
pip install -e ".[test]"     # test runner (pytest, google-re2)
pip install -e ".[vector]"   # optional ML/vector Korean name verification

The verification code loads its data (datas/, keyword/, …) relative to the project root. An editable install (pip install -e .) works out of the box. For a non-editable install where the data ships outside the package, point the engine at the data with the PII_PATTERN_ENGINE_ROOT environment variable.

Running Tests

# Run all tests
pytest

# Run re-rank scoring tests only
pytest tests/test_rerank.py -v

# Run pattern validation tests
pytest tests/test_patterns.py

# Run with coverage
pytest --cov=verification --cov-report=html

Detecting PII (end-to-end example)

A runnable reference engine ties the patterns, the Aho-Corasick pre-filter, and the verification functions together:

python examples/detect.py

import sys; sys.path.insert(0, "examples")
from detect import Detector

det = Detector()
text = "SSN 123-45-6789, email alice@example.com"
for f in det.detect_clean(text):
    print(f.category, f.text)        # -> ssn 123-45-6789  /  email alice@example.com

# The pre-filter narrows ~200 patterns to a handful before any regex runs,
# and returns identical results to scanning with every pattern:
det.detect(text, use_prefilter=True) == det.detect(text, use_prefilter=False)  # True

See examples/README.md and INTEGRATION.md.

Using the Re-Rank Scorer

import re
import yaml
from pathlib import Path
from collections import defaultdict

# Load engine config and all entity definitions
engine   = yaml.safe_load(Path('re-rank/scoring.yml').read_text(encoding='utf-8'))
entities = {}
for f in Path('re-rank').glob('*.yml'):
    if f.name == 'scoring.yml':
        continue
    data = yaml.safe_load(f.read_text(encoding='utf-8'))
    for code, cfg in data.get('entities', {}).items():
        entities[int(code)] = cfg

# Score a sentence against entity 9001 (common_email)
sentence   = "Email: alice@company.org"
entity     = entities[9001]
s_lower    = sentence.lower()

kw_hits    = [(kw, sc) for kw, sc in entity['keyword_scores'].items() if kw.lower() in s_lower]
kw_avg     = sum(sc for _, sc in kw_hits) / len(kw_hits) if kw_hits else 0
ctx_bonus  = max((b for c, b in entity['context_bonus'].items() if c.lower() in s_lower), default=0)
# regex_score added after pattern matching ...

# final_score = min(kw_avg + ctx_bonus + regex_score, 100)

See re-rank/README.md for the full scoring guide and integration examples.

Using Regex Patterns

import re
import yaml
from verification.python.verification import luhn

with open('regex/pii/common/credit-cards.yml') as f:
    patterns = yaml.safe_load(f)

visa_pattern = patterns['patterns'][0]['pattern']
text = "My card is 4111111111111111"
match = re.search(visa_pattern, text)
if match and luhn(match.group(0)):
    print("Valid Visa card detected")

Using Verification Functions

from verification.python.verification import (
    iban_mod97, luhn, high_entropy_token,
    us_ssn_valid, kr_rrn_valid, cn_national_id_valid,
    jp_my_number_valid, india_aadhaar_valid
)

iban_mod97("GB82WEST12345698765432")   # True — valid IBAN
luhn("4111111111111111")               # True — valid Luhn
kr_rrn_valid("850101-1234567")        # True — valid Korean RRN
cn_national_id_valid("11010519491231002X")  # True

Re-Rank Scoring System

The re-rank engine produces a confidence score (0–100) using three additive signals:

final_score = min(keyword_avg + context_bonus + regex_score, 100)

Signal	Description
keyword_avg	Average score of all keywords found in the text (0–100 each)
context_bonus	Flat bonus when a labeled field prefix (e.g. "SSN:") is detected
regex_score	Score of the best-matching regex pattern (calibrated 30–65)

Confidence Thresholds

Score	Label	Action
100	definite_pii	redact
80–99	high_confidence_pii	redact
60–79	probable_pii	flag
40–59	possible_pii	review
20–39	low_confidence_pii	log
0–19	not_pii	pass

Entity Coverage

Region	File	Entity Codes	Count
Korea	kr.yml	1001–1013	13
Japan	jp.yml	2001–2016	16
China	cn.yml	3001–3012	12
Taiwan	tw.yml	4001–4016	16
United States	us.yml	5001–5009	9
European Union	eu.yml	6001–6054	43
India	in.yml	7001–7009	9
Spain & France	es_fr.yml	8001–8056	12
Common / International	common.yml	9001–9013	13
Tech / Tokens	tech.yml	9501–9510	10
Total			153

See re-rank/README.md for the full scoring reference.

Regex Pattern Coverage

By Region

Region	Patterns	Key Types
Korea	30	RRN, alien reg., bank accounts, phone, vehicle, address
EU	39	National IDs, passports, VAT numbers (14 countries)
Common	25	Email, IP, credit cards, URLs, dates, names, IBAN
Taiwan	20	National IDs, bank accounts, phone, business
Japan	17	My Number, bank accounts, phone, address
Hash/Tokens	10	AWS keys, GitHub tokens, Stripe, Slack, JWT, private keys
China	12	National IDs, bank accounts, phone
India	9	Aadhaar, PAN, GST, voter ID, phone
US	10	SSN, ITIN, passport, driver's license, phone
Spain	7	DNI, NIE, CIF, phone
France	6	NIR, CNI, passport, phone
IBAN	5	DE, FR, GB, IT + generic
SOX	7	Financial control patterns
Tech	7	Patents, UUIDs, identifiers
Total	204

Data Types

• Financial: Bank accounts, credit cards, IBAN, routing numbers
• Identification: National IDs, SSN, passports, driver's licenses
• Contact: Email, phone numbers, addresses
• Network: IPv4, IPv6, URLs, MAC addresses
• Tokens/Secrets: API keys, JWT, AWS keys, GitHub tokens, private keys

Reference Registry

All 153 PII entity types have stable integer codes, safe for use in logs, pipelines, and APIs:

# reference/registry.yml (excerpt)
kr_name:                      1001
kr_rrn:                       1002
us_ssn:                       5002
common_email:                 9001
tech_aws_access_key:          9502
tech_github_token:            9504
tech_private_key:             9510

Regional detail files (reference/kr.yml, reference/eu.yml, etc.) provide per-entity metadata: severity, PII type, linked regex patterns, and keyword categories.

Verification Functions

55 functions for advanced validation beyond regex (available in Python, Go, Java, and JavaScript):

Core

Function	Purpose
iban_mod97	IBAN mod-97 checksum
luhn	Credit card Luhn algorithm
high_entropy_token	API key / secret entropy check
not_timestamp	Reject timestamp-like numbers
ipv4_public	Validate public IPv4
cjk_name_standalone	CJK standalone name format

Regional

Region	Functions
US	us_ssn_valid
Korea	kr_rrn_valid, kr_alien_registration_valid, kr_corporate_registration_valid, kr_business_registration_valid, korean_bank_account_valid
Japan	jp_my_number_valid
China / Taiwan	cn_national_id_valid, tw_national_id_valid
India	india_aadhaar_valid, india_pan_valid
Europe	spain_dni_valid, spain_nie_valid, netherlands_bsn_valid, poland_pesel_valid, sweden_personnummer_valid, france_insee_valid, belgium_rrn_valid, finland_hetu_valid

Pattern File Format

namespace: <namespace>
description: <description>

patterns:
  - id: <unique_id>
    location: <country_code>        # us, kr, jp, cn, tw, in, eu, comm
    category: <category>
    description: <description>
    pattern: '<regex_pattern>'      # single source of truth; RE2-compatible
    mask: "<redaction_format>"
    verification: <function_name>   # optional
    hit_keywords: ['AKIA', '@']     # optional; literal anchors for the
                                    # Aho-Corasick pre-filter (see prefilter/)
    min_digit_run: 4                # optional; longest consecutive-digit run every
                                    # match must contain — numeric pre-filter gate
    flags:                          # optional
      - IGNORECASE
    examples:
      match:
        - "example_match"
      nomatch:
        - "non_match"
    policy:
      store_raw: false
      action_on_match: redact
      severity: critical
    metadata:
      note: "Optional notes"
    match_type: exactly_matches     # exactly_matches | contains
    priority: 1                     # optional; lower = higher priority

Every pattern is written for cross-engine portability, so the same pattern string runs identically on Python re, Go (RE2), Java, and JavaScript:

• No engine-specific constructs — no backreferences, lookaround, inline flags, named groups, \p{...}, or POSIX classes. This keeps patterns inside the RE2 subset (enforced by tests/test_patterns.py::TestPatternRE2), which also guarantees linear-time matching (ReDoS safety) on every engine.
• Explicit ASCII character classes instead of \d and \s. \d/\s are Unicode-aware in Python (and partly in JS) but ASCII-only in Go/Java, so a bare \d would match fullwidth ０-９ or Arabic-Indic digits in Python only. Patterns use [0-9] and [\t\n\f\r ] so digit/whitespace matching is byte-for-byte identical across languages. (\b is used only next to ASCII characters, where all engines agree.)

Because the regex is identical across languages, each pattern carries a single pattern field rather than per-language copies.

What each field is for

Field	Used for	Consumed by
id	Stable identifier within the file	logs, dedup, overlap resolution
location / category	Grouping & routing (region, data type)	filtering which patterns to run
pattern	The detector itself — one RE2-safe regex	the regex engine (Stage 2)
flags	Regex flags, e.g. IGNORECASE	the regex engine
verification	Name of a checksum/validity function to confirm a raw match (Luhn, mod-97, SSA rules…)	verification/<lang>/ (Stage 3)
hit_keywords	Literal anchors for a multi-pattern pre-scan — build an Aho-Corasick automaton and skip the pattern when none are present. Pure scan optimization.	prefilter/ (Stage 1)
min_digit_run	Scan optimization for numeric patterns with no literal anchor — skip the pattern unless the text has a digit run this long.	prefilter/ (Stage 1)
mask	Redaction/replacement string once matched	masking step (Stage 4)
match_type	How a match relates to the token (whole-token vs substring); drives example validation	test suite / scanning strategy
policy	action_on_match (redact/report/…), severity, store_raw — what to do with a hit	your enforcement/DLP layer
priority	Tie-breaking when spans overlap (lower = higher priority)	match-resolution step
metadata	Human notes (provider, type)	documentation only
examples	match/nomatch fixtures	the test suite

hit_keywords and min_digit_run are optional optimization hints only — they never change what is detected, only how fast. A detector that ignores them is still correct (just slower); a detector that uses them must follow the soundness rules in prefilter/README.md. See INTEGRATION.md for the full pipeline and the contracts a consuming engine must honor.

Test Suite

File	Tests	Coverage
test_rerank.py	400	Engine config, sum_example arithmetic (153 entities), 69 sentence cases, edge cases
test_verification.py	~133	All 55 verification functions
test_patterns.py	~3,000+	YAML structure, regex compilation, match/nomatch examples
test_prefilter.py	~560	Aho-Corasick automaton, hit_keywords/min_digit_run soundness, gate behavior
test_examples.py	~16	End-to-end example; pre-filter vs brute-force equivalence
Total	3,800+

# Run all
pytest

# Re-rank only
pytest tests/test_rerank.py -v

# Filter by entity
pytest tests/test_rerank.py -k "kr_rrn or us_ssn"

# Pattern tests for a specific region
pytest tests/test_patterns.py -k "credit_card"

Development

Adding a New Pattern

1. Add the pattern to the appropriate regex/pii/<region>/ YAML file
2. Include all required fields and match/nomatch examples
3. Run pytest tests/test_patterns.py

Adding a New Re-Rank Entity

1. Add the entity block to the relevant re-rank/<region>.yml with keyword_scores, context_bonus, regex_scores, and a sum_example
2. Add the numeric code to reference/registry.yml and the regional reference/<region>.yml
3. Add sentence test cases to tests/test_rerank.py
4. Run pytest tests/test_rerank.py

Adding a Verification Function

1. Add the function to verification/python/verification.py
2. Register it in VERIFICATION_FUNCTIONS
3. Add tests to tests/test_verification.py
4. Run pytest tests/test_verification.py

License

Apache License 2.0 — see LICENSE.

Resources

• Integration Guide — how to build a detector on this library (the pipeline, what you must implement, and the contracts)
• Examples — runnable end-to-end detection using the Aho-Corasick pre-filter
• Pre-filter (Aho-Corasick + digit-run) — scan-optimization layer
• Re-Rank Scoring Guide
• Verification Functions
• Keywords
• Tests

---

Note: For legitimate use cases — data protection, compliance, and security tooling. Use responsibly and in accordance with applicable laws.