oecd_regional_education |Increasing deletion threshold

[Ashwani725]

The PR is raised to Allow up to 5% deleted records in OECD regional education validation.

Root Cause Analysis (RCA) & Threshold Justification

OECD migrated its regional education dissemination to SDMX Dataflow 2.5:

1. Eurostat NUTS 2024 Restructuring: Obsolete territorial statistical units (such as PT16, PT17, NL31, and NL33) were retired and replaced with updated boundaries (PT19–PT1D, NL35–NL36).
2. Historical Data Pruning: OECD pruned older zero-value estimates and discontinued series, resulting in 9,531 legitimate record deletions (~3.90% of total observations).

Validation Threshold Justification:

• validation_config.json sets "threshold": 5 (5%) for DELETED_RECORDS_PERCENT. This safely bounds the expected ~3.90% observation reduction caused by the upstream NUTS 2024 restructuring without triggering false pipeline failures.

---

Scope of Changes

1. Validation Configuration (validation_config.json)

• Added DELETED_RECORDS_PERCENT rule with a justified 5% threshold.
• Added DuckDB SQL date-freshness rule (check_max_date_freshness) requiring observations within date_part('year', current_date) - 3 to account for reporting lag.
• Added check_no_statvar_extinction rule asserting that all 42 StatVars are preserved and active.

2. Preprocessing Refactor (preprocess.py)

• Modularized data extraction and CSV filtering with clean error propagation (logging.error + raise).
• Pre-resolves place codes directly to DCIDs and writes unresolved codes to counters/unresolved_places.csv.
• Filters out historical non-comparable entries (STATISTICAL_OPERATION == 'SE') and drops blank observations.
• Replaced dangerous runtime file-rewrite pattern with clean file discovery and temporary file workflows.

3. Hermetic Unit Test Suite (preprocess_test.py)

• Added 11 comprehensive unit tests covering edge cases: empty files, missing headers, 100% dropped row aborts, mtime sorting, and place resolution counters.
• Completely hermetic (uses tempfile.mkdtemp(), no network/GCS calls, execution time ~0.025s).

4. Metadata & Place Mappings

• oecd_regional_education_places_resolved.csv: Added Eurostat NUTS 2024 regional codes (PT19–PT1D, NL35–NL36).
• oecd_regional_education_metadata.csv: Updated temporal coverage to 2000–2025 and updated provenance URLs.
• manifest.json: Removed deprecated custom schema MCF reference to avoid duplicate node definitions.
• test_data/: Trimmed test fixtures from ~600 rows to ~100 rows (<14 KB each) to prevent repository bloat while preserving test fidelity.

---

Test Result:

Test Cloud batch
Test GCS Bucket

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request updates the OECD regional education import manifest to include a node MCF file and a validation configuration file, and introduces a new validation configuration file to check the percentage of deleted records. The reviewer noted a mismatch in the validation configuration where the rule description specifies a 5% threshold, but the parameter is set to 4.

[saanikaaa]

I see changes are done in preprocess.py, can you pls explain the rational behind those changes

[Ashwani725]

I see changes are done in preprocess.py, can you pls explain the rational behind those changes

The changes in preprocess.py were made to optimize performance and handle the new OECD SDMX Dataflow 2.5 (OECD.CFE.EDS:DSD_REG_EDU@DF_ATTAIN(2.5)) raw CSV format (~1.08M rows and 36 columns) before passing it to stat_var_processor.py:

1. Drastic Execution Time Reduction (~6 hrs / >2 hrs on Batch down to ~12 mins):
   • Without pre-filtering and place pre-resolution, stat_var_processor.py had to parse all 1,080,385 raw rows across 36 columns and attempt place-reconciliation API lookups on ~806k unmapped/Standard Error rows, which caused execution time to take ~6 hours (and 7,561s / >2 hours on Cloud Batch).
   • By pre-filtering invalid rows and pre-resolving REF_AREA to dcid: in memory in preprocess.py, script_execution_time dropped to 588.6s (~9.8 mins) and total end-to-end Cloud Batch execution_time dropped to 703s (~11.7 mins) — well under 2 hours (>90% reduction).
2. Filter out Standard Error (SE) and empty rows:
   • The raw SDMX CSV includes both estimate values and standard error rows (STATISTICAL_OPERATION == 'SE'), as well as rows with blank OBS_VALUE. preprocess.py drops 806,350 non-observation/unmapped rows so only the 274,035 valid observation rows and the 7 required columns (REF_AREA, TIME_PERIOD, UNIT_MULT, SEX, Education level, AGE, OBS_VALUE) are processed by stat_var_processor.py.
3. Pre-resolve REF_AREA codes and log unmapped places:
   • Maps REF_AREA codes against oecd_regional_education_places_resolved.csv (including the newly added NUTS 2024 regions) directly to dcid: values, and logs any unmapped region codes to counters/unresolved_places.csv for observability.
4. Preserve raw downloaded source file & fail-fast validation:
   • Generates oecd_regional_education_data.csv while retaining the original downloaded file (A.........) in gcs_output/source_files/ (instead of os.rename() deleting it), and raises explicit exceptions (FileNotFoundError / ValueError) if the source file is missing or empty to prevent silent data drops.