Mexico_census_aa2|validation failure fix - #2140
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds validation configuration, golden data, and counters for the Mexico subnational population statistics import. Specifically, it introduces validation rules in validation_config.json, adds several golden observation and summary report CSV files, includes counter files, updates mexico_places.csv to map 'MX' to country/MEX, and updates manifest.json to integrate these validation steps and output counters. I have no feedback to provide as there are no review comments and the changes appear correct.
091daea to
cce44b2
Compare
|
Counter folder not required in Github. Please remove the folder and all file @Ashwani725 |
…o_census_aa2/counters/mexico_census_data_counters_0.csv delete counter file as not required
…o_census_aa2/counters/mexico_census_data_counters_1.csv
…o_census_aa2/counters/mexico_census_data_counters_2.csv
saanikaaa
left a comment
saanikaaa
left a comment
There was a problem hiding this comment.
Lets add a better PR description
Removed multiple GOLDENS_CHECK validation rules from the configuration.
Added header renaming and leading column definitions for standardizing dataframes. Implemented normalization function to standardize headers and column ordering for different release years.
4 participants
Summary
Fixes auto-refresh validation failure for
MexicoCensus_AA2where the national-level output (mexico_output_aa0.csv) was generated with 0 data rows.Root Cause
MXinADM0_PCODE, which was missing frommexico_places.csv(which previously only resolvedMexico->country/MEX).ADM0_ES, column index0, and accidental subnational columns in ADM0 sheets) and split senior age groups (65OrMoreYearsin 2021 into 5-year buckets up to85OrMoreYearsin 2024).Key Changes
mexico_places.csv): AddedMX->country/MEXmapping and corrected missing slash formatting in Wikidata IDs (wikidataId/Q...).mexico_download.py): Addednormalize_dataframe()to standardize headers (_ES->_EN, digit header cleanup) and column order across 2021 and 2024 releases. Moved directory creation into the download method to ensure side-effect-free module imports.mexico_download_test.py): Added test coverage fornormalize_dataframecovering ADM0, ADM1, ADM2, and non-ADM sheets.manifest.json): Configured--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf,--output_counters, and wildcardnode_mcfinputs.validation_config.json): Configured scopedSQL_VALIDATORrules verifyingMaxDate >= 2024for active StatVars andMaxDate >= 2021for legacy65OrMoreYearsStatVars.README.mdCLI examples and updatedtestdata/mexico_output_aa0.csvwith the 102 resolved observations.Verification
python3 mexico_download_test.py(6 tests passed).CL link