CaliforniaSchoolPerformance : Automation update - #2198
Open
kartik-s21 wants to merge 51 commits into
Open
Conversation
Code fix unenergy
- Add download.py to fetch and normalize multi-year CAASPP research files (2015-2019, 2021-2025) - Add PV map, metadata config, and StatVar MCF in config/ - Add manifest.json with annual cron schedule - Add golden summary report in golden_data/ - Add unit tests in test_data/ - Add README documentation and run_download_process.sh
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces an automated Data Commons import pipeline for California School Performance (CAASPP) academic data, including download scripts, configuration mappings, orchestration bash scripts, and unit tests. The review feedback highlights several areas for improvement: adding robust error handling in download.py to raise exceptions on failed downloads and prevent silent pipeline failures, wrapping year-range parsing in try-except blocks to handle malformed CLI inputs, making the state/county entity filtering logic more resilient to padding variations, and removing redundant individual-year processing in run_download_process.sh to optimize execution time.
- Add validation_config.json with 0.1 deletion threshold and golden check rules - Add golden_observations.csv generated via validator_goldens.py against top 100k places - Update golden_summary_report.csv with standardized stable schema property sets - Add counters/california_school_performance_counters.csv and configure in manifest.json - Register node_mcf in manifest.json for generated MCF staging to GCS
… filtering, and script optimization
- Raise RuntimeError if no files were successfully downloaded in download.py
- Add try-except to handle malformed year range format in parse_years
- Make state/county entity filtering robust to zero-padding variations using lstrip('0')
- Remove redundant individual-year processing loop in run_download_process.sh
… data, and harden pipeline script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Converts the
CaliforniaSchoolPerformanceimport from a legacy manual script to an automated StatVar import with multi-year data support (2015–2019, 2021–2025; 2020 was cancelled due to COVID-19).Key Changes
download.py: Downloads and unifies CAASPP research files across three different historical format eras (comma-delimited 2015–2019, caret-delimited 2021–2023, and updated 69-column caret format 2024–2025). Filters for state (00000/0000000) and county levels.config/): PV mapping, metadata configuration, and StatVar MCF covering grades 3–8, 11, 13 for ELA & Math across 55 demographic subgroups.manifest.json): Configured with an annual cron schedule (0 0 1 10 *) matching the CAASPP October release cadence.test_california_school_performance.pyandgolden_summary_report.csv.README.mdandrun_download_process.shscript.Verification & Testing
python3 -m unittest test_data/test_california_school_performance.pyRelated CL
Internal Google3 CL updating
.textprotomanifest, provenance.mcf, and import group test.Code Review And Troubleshooting Report