Skip to content

docs(methodology): add Bertrand-Duflo-Mullainathan (2004) paper review#553

Merged
igerber merged 1 commit into
mainfrom
docs/placebo-bdm-2004-review
Jun 26, 2026
Merged

docs(methodology): add Bertrand-Duflo-Mullainathan (2004) paper review#553
igerber merged 1 commit into
mainfrom
docs/placebo-bdm-2004-review

Conversation

@igerber

@igerber igerber commented Jun 26, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add a paper-sourced methodology review of Bertrand, Duflo & Mullainathan (2004), How Much Should We Trust Differences-in-Differences Estimates?, QJE 119(1):249-275 — docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md.
  • Grounds the PlaceboTests diagnostic in BDM's placebo-law / randomization-inference framework: the two-step random assignment (uniform fake date 1985-1995; 25 of 50 random treated groups), the serial-correlation over-rejection evidence (Table II: 67.5% uncorrected, 44% state-year cluster, 5% serially-uncorrelated), and the four inference corrections BDM evaluate (parametric AR, block/percentile-t bootstrap, state-level cluster VCV, time-series aggregation).
  • Captures the two load-bearing footnotes for the permutation diagnostic: fn 11 (randomization-inference link) and fn 12 (the placebo distribution is not standard normal, so the empirical distribution is the correct reference).

This is the paper-review half (PR-A) of the PlaceboTests methodology promotion. Integrating the proposed ## PlaceboTests registry entry into REGISTRY.md and promoting the METHODOLOGY_REVIEW.md tracker row from In Progress to Complete are the work of the follow-up source-validation pass; the new file marks its registry entry as proposed and not yet integrated.

Methodology references

  • Method name(s): PlaceboTests (placebo-law / randomization-inference diagnostic; serial-correlation DiD inference)
  • Paper / source link(s): Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How Much Should We Trust Differences-in-Differences Estimates? QJE 119(1), 249-275. https://doi.org/10.1162/003355304772839588
  • Any intentional deviations from the source (and why): None — this PR adds a paper review only. The review faithfully distinguishes BDM-derived requirements from library extensions (leave_one_out, chosen-pre-period timing test, finite-draw p-value floor) and labels the diff-diff NaN-inference convention as a defensive enhancement, not a BDM requirement.

Validation

  • Tests added/updated: No test changes (paper-review documentation only).
  • Backtest / simulation / notebook evidence (if applicable): N/A — no executable behavior changed.

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

Overall Assessment

✅ Looks good

No unmitigated P0/P1 findings. The PR is documentation-only and the added BDM review is explicitly marked as proposed, not an authoritative REGISTRY.md change.

Executive Summary

  • No estimator, weighting, SE, variance, default, or executable behavior changed.
  • The BDM scope is correctly framed as inference/diagnostics rather than a DiD estimator; the published abstract aligns with that framing. (doi.org)
  • The proposed PlaceboTests entry does not conflict with the current registry stub because it is clearly non-integrated.
  • Two P2 documentation/process issues should be cleaned up before relying on this as tracker-grade methodology evidence.

Methodology

  • Severity: P3-informational
    Impact: The proposed registry text is not yet authoritative. docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L22-L28 says the entry is proposed, while docs/methodology/REGISTRY.md:L3098-L3107 remains the current edge-case-only PlaceboTests contract. This is acceptable for this PR.
    Concrete fix: No action required in this PR; validate and integrate into REGISTRY.md in the follow-up promotion pass.

Code Quality

No findings. No code changed.

Performance

No findings. No runtime paths changed.

Maintainability

  • Severity: P2
    Impact: The methodology tracker is now stale. METHODOLOGY_REVIEW.md:L1312-L1329 still says PlaceboTests has no canonical primary reference and only registry/tests documentation, and METHODOLOGY_REVIEW.md:L1464-L1466 still lists it as missing a paper review. That makes the new review hard for future source-validation work to discover.
    Concrete fix: Update the PlaceboTests tracker row/details to reference BDM (2004) and docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md, while keeping status In Progress if registry integration remains deferred.

Tech Debt

No untracked correctness debt introduced. The deferred registry/promotion work is called out in the new review at docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L22-L24; updating METHODOLOGY_REVIEW.md would make that tracking explicit.

Security

No findings. No secrets or executable surfaces added.

Documentation/Tests

  • Severity: P2
    Impact: docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L5 cites papers/119-1-249.pdf, but that file is not present in the repo. Other paper reviews usually state that local PDFs are gitignored and name the DOI/landing page as authoritative. This hurts reproducibility of the page/table checks.
    Concrete fix: Reword the PDF reviewed line to follow the project convention, e.g. published QJE PDF via DOI, local PDFs gitignored under /papers/, DOI version authoritative.

No test changes are needed for this docs-only PR.

PR-A of the PlaceboTests methodology promotion. Adds a paper-sourced review
of "How Much Should We Trust Differences-in-Differences Estimates?"
(QJE 119(1):249-275) grounding the diff-diff PlaceboTests diagnostic in the
placebo-law / randomization-inference framework, the serial-correlation
over-rejection problem, and the four inference corrections BDM evaluate.

Updates the METHODOLOGY_REVIEW.md PlaceboTests tracker row to reference
BDM (2004) and the new review (status stays In Progress; REGISTRY integration
and promotion remain for the source-validation pass).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@igerber igerber force-pushed the docs/placebo-bdm-2004-review branch from f50e3b4 to bcc4341 Compare June 26, 2026 18:51
@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: bcc434139fd46e34c1c6526cb54e1a468867f627


Overall Assessment

✅ Looks good

No unmitigated P0/P1 findings. This is documentation-only, and the two prior P2 review items are addressed.

Executive Summary

  • No estimator, weighting, variance/SE, inference, assumptions, or defaults changed.
  • The affected method surface is PlaceboTests only, as documentation/methodology tracking.
  • METHODOLOGY_REVIEW.md:L85 and METHODOLOGY_REVIEW.md:L1317-L1330 now reference BDM (2004) and track the remaining registry/test promotion work.
  • The new review clearly marks its richer ## PlaceboTests registry text as proposed, not authoritative, at docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L22-L28.
  • High-level BDM framing matches the DOI record: placebo laws, serial-correlation-driven SE understatement, and evaluated corrections via bootstrap, asymptotic VCV, and time aggregation. (doi.org)

Methodology

  • Severity: P3-informational
    Impact: The proposed PlaceboTests registry entry is intentionally not integrated yet. The current authoritative registry remains the edge-case stub at docs/methodology/REGISTRY.md:L3098-L3107, while the tracker explicitly leaves integration and a dedicated methodology test file as outstanding at METHODOLOGY_REVIEW.md:L1327-L1330. This is documented deferred work, not a methodology defect.
    Concrete fix: No action required in this PR; complete the listed follow-up by integrating the proposed entry into REGISTRY.md and adding tests/test_methodology_placebo.py.

Code Quality

No findings. No code changed.

Performance

No findings. No runtime paths changed.

Maintainability

No findings. The prior tracker staleness issue is resolved by the updates at METHODOLOGY_REVIEW.md:L1317-L1330 and METHODOLOGY_REVIEW.md:L1468.

Tech Debt

No untracked correctness debt introduced. Remaining PlaceboTests promotion work is explicitly tracked in METHODOLOGY_REVIEW.md:L1327-L1330.

Security

No findings. The PR adds methodology markdown only; no secrets or executable surfaces were introduced.

Documentation/Tests

No blocking findings. The prior PDF reproducibility issue is resolved at docs/methodology/papers/bertrand-duflo-mullainathan-2004-review.md:L5, which now states the DOI/journal version is authoritative and local PDFs are gitignored.

Validation: git diff --check is clean. No tests were run because this PR changes documentation only.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Jun 26, 2026
@igerber igerber merged commit 606318c into main Jun 26, 2026
11 of 12 checks passed
@igerber igerber deleted the docs/placebo-bdm-2004-review branch June 26, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant