Skip to content

Campaign finance scrapper for legislators#2195

Open
J-C-L wants to merge 10 commits into
codeforboston:mainfrom
J-C-L:finance-scrapper
Open

Campaign finance scrapper for legislators#2195
J-C-L wants to merge 10 commits into
codeforboston:mainfrom
J-C-L:finance-scrapper

Conversation

@J-C-L

@J-C-L J-C-L commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

This populates the finance tab on the legislator profile pages. See Issue #2118
It includes:

  • Fetches campaign finance data from the OCPF website [currently this is a run on demand function. Once data is validated, we'll switch to nightly scheduled run
  • Stores campaign finance data per legislator in Firestore under new collection: generalCourts > 194 > membersFinance
    <Member_ID>
  • Displays campaign finance summary on finance tab of legislator page for each legislator

Checklist

  • [Y] On the frontend, I've made my strings translate-able.
  • [N/A ] If I've added shared components, I've added a storybook story.
  • [Y ] I've made pages responsive and look good on mobile.
  • [ N/A] If I've added new Firestore queries, I've added any new required indexes to firestore.indexes.json (Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)

Screenshots

finance-tab-screenshot

Known issues

  • scrapeOcpfFinance function must be run after matchOcpfMembers, so court member IDs are already associated in firestore with OCPF IDs
  • Code is safe to push while legislator pages are behind a flag, and hidden in prod.
  • Still need to consult with campaign finance expert on some data choices made here.
  • "small donors" box is not yet populated. Need to decide what we want to use there (<$50 or <$200)

Steps to test/reproduce

For reproducing locally until firebase functions deployed:

  1. Load members in firestore (run a few times with delay)
    curl "http://localhost:5001/demo-dtp/us-central1/triggerPubsubFunction?scheduled=startMemberBatches"

  2. Match member IDs:
    curl -X POST http://localhost:5001/demo-dtp/us-central1/matchOcpfMembers

  3. Run finance scraper:
    curl -X POST http://localhost:5001/demo-dtp/us-central1/scrapeOcpfFinance

  4. View Legislator Page:
    Ex: http://localhost:3000/legislators/194/RLR0

@vercel

vercel Bot commented Jul 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
maple-dev Building Building Preview, Comment Jul 21, 2026 10:49pm

Request Review

@Mephistic Mephistic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - this is fantastic! Love the detailed breakdown comments on the data wrangling - those are going to be so handy for maintaining this pipeline going forward.

I know we still have unresolved questions that we're waiting on the subject matter expert for answers, but I'm fine merging this and building on top of it if anything radically changes based on that feedback.


const OCPF_BASE_URL = "https://ocpf2.blob.core.windows.net/downloads/data2"

const YEARS = ["2025", "2026"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to start running this nightly, should we tie this to the years associated with the currentGeneralCourt for the next legislative session (or just use the current year and previous year)?

…tribution Breakdown, as well as dates each are calculated through
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants