Minnesota state dataset: Population/household undercount and multi-tax-unit household concentration bug

[DTrim99]

Summary

Analysis of the Minnesota (MN) state dataset reveals significant discrepancies from actual Minnesota statistics and a data quality issue with multi-tax-unit household distribution.

Issues Found

1. Population and Household Undercount

Metric	MN Dataset	Real/Target	Difference
Population	~4.1M	5.74M	-29%
Households	1,254,857	~2,344,432	-46%

Sources:

• Census Bureau QuickFacts: Minnesota
• Census Reporter - Minnesota

2. Multi-Tax-Unit Households Concentrated in Top Income Deciles

When analyzing a CTC reform that should only affect low-income households (phases out at ~\0-40k), we found unexpected impacts in the 8th, 9th, and 10th income deciles.

Investigation revealed that affected high-income households have dramatically more tax units than unaffected ones:

Metric	Affected Top-Decile HH	Unaffected Top-Decile HH
Avg tax units per household	6.41	1.55
Household count	50,907	325,752

Distribution of tax units among affected top-decile households:

• 53% have 8 tax units
• 40% have 5 tax units
• Only 6% have 2 tax units

This is likely a bug - multi-tax-unit households should be distributed across the income spectrum, not concentrated in the top deciles.

Impact

This causes misleading results when analyzing policies that target low-income populations (like CTCs, EITC, etc.), as the impacts appear to affect wealthy households when they shouldn't.

Investigation Details

Full analysis documented in: PolicyEngine/analysis-notebooks#108

Suggested Fix

Review the household/tax-unit mapping and weighting in the state dataset calibration to ensure:

1. Population and household counts match targets
2. Multi-tax-unit households are distributed realistically across income deciles

[PavelMakarchuk]

Detailed Diagnostic Analysis

I ran a comprehensive diagnostic on the MN.h5 dataset from HuggingFace. Here are the specific findings:

1. Calibration Target Validation

The dataset is significantly missing state-level calibration targets:

Metric	Simulated	Target	Deviation
Population	3,985,277	5,737,915	-30.5%
Households	1,229,850	2,344,432	-47.5%
Tax Units	2,081,321	2,871,840	-27.5%

2. AGI Distribution Issues

The AGI distribution is severely distorted - over-representing extremes while under-representing middle brackets:

AGI Bracket	Simulated TUs	Target TUs	Deviation
<$1	108,997	10,930	+897%
$500k+	132,455	38,950	+240%
$75k-$100k	93,814	288,810	-67.5%
$50k-$75k	173,671	403,640	-57%
$40k-$50k	75,936	179,700	-57.7%
$100k-$200k	293,167	514,790	-43.1%

3. Weight Distribution Analysis

The weight distribution is extremely concentrated:

• 68.2% of household records have weight < 1 (contributing only 0.4% of total weight)
• 1% of records (664 households) account for 62% of total weight
• Top 5% of households by weight contribute 83% of total weight

4. Most Impactful Household Identified

I identified the specific household causing the largest CTC distortion:

• Household AGI: $622,863
• Structure: 5 tax units (3 adults aged 60, 29, 29 + 2 children)
• Replication: Appears in all 8 MN congressional districts
• Total weight: ~20,000
• CTC contribution: $159M weighted (9% of MN's total $1.75B CTC)

This single household structure, replicated 8 times with high weights, accounts for most of the "CTC affecting rich households" effect.

5. Root Cause Assessment

The evidence points to multiple interacting issues:

1. CD-stacking weights don't aggregate to state totals - The sum of household weights across congressional districts (~1.23M) is far below the state household target (~2.34M)

2. Extreme weight concentration - The calibration process is concentrating weight on a small subset of households rather than distributing it across the sample

3. Multi-TU amplification - When a multi-TU household gets high weight AND gets replicated across all CDs, its quirks (like having low-income TUs within a high-AGI household) get amplified dramatically

4. Missing middle-income representation - The calibration is not achieving the correct AGI distribution, with middle brackets severely underweighted

Diagnostic Code

I've created diagnostic tools in policyengine_us_data/diagnostics/ that can be used to validate any state dataset:

• mn_calibration_diagnostic.py - MN-specific multi-TU analysis
• calibration_benchmarks.py - General benchmark validation against SOI targets

Suggested Investigation Path

1. Review how CD-level calibration targets are set and whether they're constrained to sum to state totals
2. Investigate why the weight calibration is concentrating on so few records
3. Check if multi-TU households are being handled specially (or should be) in the stacking algorithm
4. Validate the source CPS data to ensure multi-TU households aren't over-represented at baseline

[baogorek]

Reweighting with more regularization helped the Rhode Island reform a lot. The weights are all over the place right now. @DTrim99 and @PavelMakarchuk , I'm going to be refitting all weights and new subsampling method today and we'll see if this persists.

[baogorek]

Hopefully the fix that's coming fixes this

● Synthesis                                                                                                                                                   
  ┌───────────────────────┬──────────────────────────┬───────────────────┬─────────────────────┐                                                              
  │        Metric         │ HuggingFace MN (Problem) │  Local MN (Fix)   │ National (Baseline) │                                                              
  ├───────────────────────┼──────────────────────────┼───────────────────┼─────────────────────┤                                                              
  │ Population            │ 4.03M (-30%)             │ 5.74M (on target) │ 336M                │                                                              
  ├───────────────────────┼──────────────────────────┼───────────────────┼─────────────────────┤                                                              
  │ Households            │ 1.24M (-47%)             │ 1.94M (-17%)      │ 144M                │                                                              
  ├───────────────────────┼──────────────────────────┼───────────────────┼─────────────────────┤                                                              
  │ Decile 10 avg AGI     │ $1,231,274               │ $528,338          │ $553,987            │                                                              
  ├───────────────────────┼──────────────────────────┼───────────────────┼─────────────────────┤                                                              
  │ Decile 8 tax units/HH │ 3.36                     │ 1.57              │ 1.34                │                                                              
  ├───────────────────────┼──────────────────────────┼───────────────────┼─────────────────────┤                                                              
  │ CTC D8→D9 reversal    │ 10.3%→26.6% (yes)        │ 55.3%→51.2% (no)  │ 37.1%→41.4% (no)    │                                                              
  └───────────────────────┴──────────────────────────┴───────────────────┴─────────────────────┘                                                              
  What Happened (The Problem)                                                                                                                                 
                                                                                                                                                              
  The old MN calibration concentrated weights on multi-tax-unit households. Evidence:                                                                         
  1. Decile 8 averages 3.36 tax units/household (vs ~1.5 normal)                                                                                              
  2. Household AGI is inflated because it sums AGI across all tax units                                                                                       
  3. This creates artificial undercount - weights piled onto few records instead of spreading                                                                 
                                                                                                                                                              
  The CTC reversal (decile 8: 10.3% → decile 9: 26.6%) happens because these multi-TU households contain both high-AGI tax units (pushing household to high   
  decile) and low-AGI tax units (qualifying for CTC). The uneven weight distribution makes this pattern visible.                                              
                                                                                                                                                              
  The Fix Works                                                                                                                                               
                                                                                                                                                              
  Your local MN dataset:                                                                                                                                      
  - Nails the population target (5.74M vs 5.74M)                                                                                                              
  - Smooth tax unit distribution (~1.4-1.8 across deciles, no spikes)                                                                                         
  - Reasonable AGI (decile 10 at $528k, similar to national)                                                                                                  
  - No CTC reversal (monotonic decrease in upper deciles)                                                                                                     
                                                                                                                                                              
  The remaining household gap (-17%) may be a data limitation rather than a calibration bug.    

[baogorek]

Closing after republishing. https://github.com/PolicyEngine/policyengine-us-data/actions/runs/21259987859