Stop the US PUF imputation from predicting income from itself (#982)

[MaxGhenis]

Fixes #982.

Summary

The US PUF tax-detail QRF predicted each income item from the recipient's own survey value of that item. On the donor side, each of those predictor columns was an alias of the donor's own imputed output, so the forest learned output = input. On the 2026-09-12 genuine-build inputs, PUF-clone wages had rank correlation 1.000 with survey wages. Among units with positive survey wages, 99.98% landed within 10% of the survey value. No clone exceeded the survey maximum, so ASEC topcodes and underreporting carried straight into the PUF half. This is the root of the missing $5M+ tail in #958.

This PR replaces the six survey income levels with these predictors, derived identically on both sides:

• Demographics. Head age, spouse age (0 without a spouse), head sex and dependent count come from person age, sex and tax-unit role. The survey side reads its person table and the PUF side reads the processed PUF person arrays.
• Income rank share. Each unit's weighted mid-rank of the six-item income total within its own population: the PUF-detail recipient rows on the survey side, the donor table on the PUF side.
• Has earnings. Set when wages or self-employment income is nonzero. The PUF covers filers only, so without this flag the rank handed wages to 48% of survey non-earners.

Filing status and unit size stay. No income level is a predictor any more.

Evidence (pre-calibration)

The fit uses the saved 9/12 donor frame (211,677 tax units), predicted onto 231,007 PUF-clone recipients at 8 trees per variant. AGI is a proxy: the sum of 15 imputed items. Full receipts are in experiments/982-puf-self-prediction/.

Returns by proxy AGI	Old	New	SOI Table 1.1 (TY2023)
$1.5M–2M	199,843	147,280	147,290
$2M–5M	200,741	213,125	203,229
$5M–10M	23,600	48,629	49,262
$10M+	8,186	33,920	30,382

Wage measure	Old	New
Rank correlation with survey wages	1.000	0.906
Within 10% of survey value (units with positive survey wages)	99.98%	24.0%
Survey non-earner units given any earnings (of 67,908)	127	7
Wage-positive share (survey 67.7%)	67.7%	67.5%

Capital gains are 42.6% of $10M+ AGI under the new design, against 39.5% in SOI Table 1.4 and 0.1% under the old one.

Invariants

These hold for every input, and Hypothesis property tests in test_us_puf_income_rank_properties.py check them:

• Every rank share lies strictly between 0 and 1 when weights are positive.
• The weighted mean share is exactly one half.
• A higher value never gets a larger share, and tied values share one rank.
• Row order and weight scale do not change any rank, so clone copies that split a household's weight rank identically.
• Rows with a missing value are excluded and do not move anyone else's rank.
• The vectorized rank equals the pairwise definition. This is the differential test.
• The earnings flag is 1 exactly when a component is nonzero, and missing when any component is missing.

Structurally, no default predictor resolves to an imputed output column. test_default_predictors_never_resolve_to_an_imputed_output pins this so the self-prediction cannot come back.

Fail-closed demographics

The demographics refuse malformed input rather than filling it:

• Survey side. A tax-unit role outside HEAD, SPOUSE and DEPENDENT raises. A missing role leaves that unit's demographics missing under the strict policy, so the completeness check names it, and raises under the legacy policy.
• PUF side. A missing age, sex or role flag raises, as does a person with no role, is_female and is_male that disagree, or a unit without a head. The processed PUF has none of these: 499,045 persons, no missing values, one role each, one head per unit. Neither does the 9/12 survey frame.

Tests

• New predictor tests. test_us_puf_income_rank_predictors.py covers the mid-rank, demographics, donor construction from PUF arrays, recipient-row scope, the strict role check, and two end-to-end discriminating cases:
  • Topcoded survey units receive PUF wages above the topcode, and the old predictors do not.
  • Survey retirees receive no wages with the flag and do receive them without it.
• Existing tests updated. Predictor counts and fixtures were updated, and the US spec was regenerated and re-pinned on the merge with main: predictor tuples, resource semantics, checkpoint identity, seed protocol and map, and coverage evidence.

Not in this PR

• Colorado (State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940) is not fixed by this alone. One survey-weighted record draws $125.4M and carries 84% of Colorado's pre-calibration income above $1M (receipted in scores_8_trees.json). State top-tail totals stay noisy until the state × AGI-band amount targets from State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 enter calibration.
• The tail and targets PRs still pair with this. Bind the SOI Table 1.1 size-of-AGI distribution as calibration targets #960 (Table 1.1 targets) and Carry PUF high-AGI donors into the own-tail stratum with their full income vector #964 (full-vector tail) are separate. This PR fixes the clone-1 bulk and the $1M–$10M range.
• The graph-native path is untouched. The native full_puf_enrichment path imports this predictor list and will pick up the new predictors. Its owning session was told in the coordination file.

axiom: n/a: microsimulation data imputation, no policy rule changes.

🤖 Generated with Claude Code

[MaxGhenis]

Independent review by GPT-6 Astra through Subfleet, read-only.

Round 1 (head 4a49cfcbe) requested changes. It raised two major findings:

• Unrecognized survey roles were silently ignored.
• Missing PUF demographics were zero-filled, so a missing sex became male or female depending on the array.

It also raised three minor findings about evidence accuracy. ddd5f6e38 fixes all five, as listed in the commit message and the PR description.

Round 2 (head ddd5f6e38) approved, with all five findings verified resolved. It raised two low-priority cleanups:

• A comment overstated that the earnings flag prevents earnings for non-earners.
• A role constant was declared twice.

8ea1d24f2 applies both cleanups and re-pins the digests that hash puf_support.py source bytes: the seed protocol and seed map, the US bundle digest, and the loader golden. It makes no other change.

[MaxGhenis]

CI on 8ea1d24f2 failed four pool tests (test_us_multispine_pool.py) in the us-am, fast-rest and wheels lanes. The failures had one cause: the hand-built ACS fixture had no tax_unit_role_input column, and the fail-closed demographics added in round 1 correctly refused its two role-less people. Production ACS spines get the role from assign_us_unit_structure. 57216feeb gives the fixture that role, makes the error name the affected tax units, and re-pins the source-hashed digests. The four tests pass locally. There is no change to imputation logic since the approved head.

[MaxGhenis]

Correction to my earlier numbers. The top-tail figures I posted weighted the PUF-clone half ×2, "as if this half were the whole population", and compared that with SOI. The real pool gives the survey half the other 50% of the weight, and the survey half has no units at $5M or more. Both halves at frame weight, before calibration and before main's capital-gains tail stage (8 trees, 15-item proxy AGI):

Band	Old design	#1033	SOI Table 1.1
$1M–1.5M returns	701k (1.90×)	483k (1.31×)	369k
$5M–10M returns	11.8k (0.24×)	24.3k (0.49×)	49.3k
$10M+ returns	4.1k (0.13×)	17.0k (0.56×)	30.4k
$10M+ AGI	$92B	$519B	$908B
Colorado income above $1M	$20.2B	$66.3B (2.1×)	$31.7B

#1033 roughly quadruples the pool's mass at $5M+, but on its own it reaches about half of SOI there, not SOI. Colorado is about 2× SOI above $1M, not 4×. One record still carries 81% of that. The script and output are at ~/PolicyEngine/_recovered/lane958-958-964-982-archive-20260926/982/actual_pool_bands.{py,json}. A follow-up will add them to experiments/982-puf-self-prediction/.