[crdc_instructional_wifi_devices] Modified pvmap

[Krishnam24maheshwari]

Summary

Fixes import validation failures and hardens the data pipeline for crdc_instructional_wifi_devices to support 2024 CRDC survey data. Updates column mappings for SCH_JUST, expands metadata column width to 15, quotes manifest input globs, enables telemetry counters, passes generated node MCFs, updates test fixtures, hardens download.py for dynamic multi-year ingestion, and configures validation_config.json with date freshness and consistency rules.

Bug: b/565616431
Related CL: cl/983104185 (ProvisionalNodePlaces.mcf registering the 69 new 2024 CRDC NCES schools)
PR: #2237

---

Root Cause Analysis

1. Missing Reference on StatVars:
   manifest.json omitted "node_mcf": "output/*.mcf" under import_inputs. Consequently, generated StatVar MCF definitions were not passed to genmcf, leading to reference check failures against unindexed StatVars.
2. Column Renaming in 2024 Data (JJ → SCH_JUST):
   In the 2023–24 CRDC release (Internet_Access_and_Devices_2024.csv), CRDC renamed the juvenile justice facility column from JJ to SCH_JUST and positioned it beyond column 8. Because common_pvmap.csv only mapped JJ and common_metadata.csv hardcoded mapped_columns,8, the juvenile justice property was dropped for 2024 observations.
3. Pipeline Fragility in download.py:
   The download script used bare requests.get() without session reuse or retries, lacked timeouts, buffered multi-hundred megabyte zip files directly into RAM, performed unbounded zip extractions, swallowed download exceptions (exiting with code 0 on complete failure), and did not preserve leading zeros in NCES identifier codes (COMBOKEY, LEAID, SCHID).
4. Missing Telemetry Counters, Validation Config, and Operational Artifacts:
   The processor script did not emit --output_counters, and manifest.json lacked a custom validation_config_file, defaulting to zero-tolerance thresholds. In addition, validation_config.json was omitted from source_files, preventing it from being archived during automated batch runs.

---

Changes Made

1. Column Mapping & Schema Alignment

• common_pvmap.csv:
  • Added mappings for the 2024 column SCH_JUST:

    SCH_JUST:Yes,educationalInstitutionType,JuvenileJusticeFacility,,,,,,
    SCH_JUST:No,educationalInstitutionType,NotJuvenileJusticeFacility,,,,,,

  • Preserved backward compatibility for 2020–21 data using column JJ.
• common_metadata.csv:
  • Increased mapped_columns from 8 to 15 to ensure all 14/15 columns in the 2024 schema are fully processed across all survey years.

2. Pipeline Hardening & Resilience (download.py)

• Connection Pooling & Retries: Replaced bare requests.get() with requests.Session() and an exponential backoff HTTPAdapter with Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504]).
• Request Headers & Timeouts: Added browser User-Agent and Accept headers to prevent CDN/CloudFront blocks, and set explicit 60-second timeouts.
• Dynamic Chronological Ingestion: Dynamically scans academic years starting from 2020–21 through the latest available year, gracefully skipping biennial gap years (e.g. 2022–23) without premature loop breaks.
• Fail-Loud Error Handling: Differentiates HTTP 404 (skip unreleased/gap years) from 5xx/network errors (re-raise). Exits with a non-zero status code (sys.exit(1)) if no datasets are successfully downloaded.
• Data Integrity (NCES IDs): Added dtype=str to pd.read_csv() to prevent truncation of leading zeros in COMBOKEY, LEAID, and SCHID.
• Atomic File Writes: Writes DataFrames to .csv.tmp first, verifies file size, and atomically replaces the target file using .replace().
• Secure Selective ZIP Extraction: Streams ZIP archives to temporary disk and selectively extracts only the single target CSV using safe basenames, mitigating Zip Slip vulnerabilities.
• Academic Year Parsing: Replaced IndexError fallback with strict regex validation (r"^(\d{4})-(\d{2})$"), raising ValueError on malformed years rather than emitting year 0.
• Code Hygiene: Anchored OUTPUT_DIR relative to script location (Path(__file__).resolve().parent / "input_files"), added Apache 2.0 license header, removed unused imports, and formatted lines to <= 100 characters.

3. Manifest Configuration (manifest.json)

• Node MCF: Added "node_mcf": "output/*.mcf" to import_inputs to ensure generated StatVars are resolved.
• Quoted Input Glob: Single-quoted --input_data='input_files/*.csv' to prevent shell glob expansion.
• Telemetry: Added --output_counters=counters/instructional_wifi_devices_counters.csv to the processor script command and added "counters/*.csv" to source_files.
• Validation Artifact Archiving: Added "validation_config.json" to source_files and configured "validation_config_file": "validation_config.json".

4. Validation Rules (validation_config.json)

• Date Freshness: Added check_latest_date_freshness using SQL_VALIDATOR (latest_year >= 2024) to assert observations reach the latest 2024 CRDC release.
• Date Consistency: Added check_max_date_consistent (MAX_DATE_CONSISTENT) across both StatVars.
• Deletion Threshold: Configured check_deleted_records_percent with "threshold": 0.1 and an explicit description to account for biennial school churn.
• The 0.1% record deletion threshold accommodates natural biennial entity attrition (school closures, consolidations, and NCES ID retirements) across ~100,000 U.S. public schools.

5. Test Data & Documentation

• test_data/:
  • Trimmed sample input Internet_Access_and_Devices_2021.csv to 51 rows (<100 rows).
  • Removed stale internet_access.* fixtures and regenerated manifest-aligned instructional_wifi_devices.csv and instructional_wifi_devices.tmcf.
• README.md:
  • Documented automated NCES place resolution (nces/{Data:0>12}).
  • Listed canonical StatVar DCIDs (Count_School_JuvenileJusticeFacility_WifiEnabledDevice and Count_School_NotJuvenileJusticeFacility_WifiEnabledDevice).
  • Documented biennial release cadence (P2Y), step-by-step execution commands, and validation rules.

---

Verification & Test Results

The import was tested via Cloud Batch in datcom-infosys-dev:

• GCS Run Directory:
  https://console.cloud.google.com/storage/browser/datcom-import-test/statvar_imports/crdc/instructional_wifi_devices/crdc_instructional_wifi_devices/2026_09_25T01_02_11_528535_07_00?project=datcom-infosys-dev

• Validation Output (validation_output.csv):
  https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/crdc/instructional_wifi_devices/crdc_instructional_wifi_devices/2026_09_25T01_02_11_528535_07_00/input0/validation/validation_output.csv?project=datcom-infosys-dev

• Differ Generated Locally:
  https://console.cloud.google.com/storage/browser/unresolved_mcf/crdc_differ?project=datcom-204919

[gemini-code-assist]

Code Review

This pull request updates the instructional Wi-Fi devices import configuration, including updating the README, adding a validation configuration file, and updating the manifest to output counters and include provisional MCF files. However, a critical issue was identified in common_pvmap.csv where the header row was modified to remove the column names p1, v1, p2, v2, which are required by the PropertyValueMapper and should be reverted.