improving lobbying data pipeline reliability - #2

Closed

nesanders wants to merge 132 commits into

mainfrom

lobbying-pipeline-reliability

Commits on Mar 25, 2026

fix: redirect logged-out users to login when clicking Follow button
slarson
committed
34b3157

Commits on Jun 4, 2026

initial plan
nesanders
committed
0348b29
feat: add lobbying disclosure ingestion pipeline

nesanders
and
claude
committed
774568e

Commits on Jun 5, 2026

feat: add Python Cloud Run scraper for lobbying disclosures

nesanders
and
claude
committed
0074294

Commits on Jun 9, 2026

refactor: remove dead TypeScript scraper code from lobbying module

nesanders
and
claude
committed
2bcb783

Commits on Jun 16, 2026

Link legislator profiles to district pages
Smoss
committed
3689796
Enhance SEC district parsing and testing
Smoss
committed
f0aacd5

Commits on Jun 17, 2026

Add feature flag check for legislator profile page
Smoss
committed
253014e
Refactor district parsing tests and update project configuration
Smoss
committed
9e8a14f

Commits on Jun 26, 2026

Delete .tool-versions (#2173)
nesanders
authored
beb1861
Merge remote-tracking branch 'upstream/main' into lobbying-data-pipeline
nesanders
committed
df7c78e

Commits on Jun 28, 2026

feat: parser fixes for all 4 disclosure eras + GCS archiving infrastructure

nesanders
and
claude
committed
26150ef

Commits on Jun 29, 2026

fix: Firestore write path bugs found during dev write test

nesanders
and
claude
committed
b66576a
docs: update lobbying ingestion doc to reflect Python scraper and all 4 HTML eras

nesanders
and
claude
committed
6129130

Commits on Jun 30, 2026

docs: add real Firestore examples and fix registrantId description

nesanders
and
claude
committed
1ef618a

Commits on Jul 1, 2026

Cross-compatibility (#2174)
delexagon
authored
f43d5b0
moar Homepage Tweaks (#2172)
mertbagt
authored
c28e6a1

Commits on Jul 2, 2026

Merge pull request #2145 from Smoss/smoss/scrape-ma-district-pages
mertbagt
authored
e6c3fb3
update browser tab title
mertbagt
committed
1f39cfb
directory path
mertbagt
committed
90f9273

Commits on Jul 3, 2026

legislator header refactored
mertbagt
committed
56e8603

Commits on Jul 4, 2026

legislator tabs
mertbagt
committed
9718b06
refactor sidebar
mertbagt
committed
81ecbf9
tab containers
mertbagt
committed
2e18f8c
resposive margings
mertbagt
committed
f4780c1

Commits on Jul 7, 2026

fix: add getStaticProps to token page and wire prod MCP env (#2176)

nesanders
and
claude
authored
eae3474
added translations
mertbagt
committed
15543fd
Merge pull request #2180 from mertbagt/refactor-legis-header
mertbagt
authored
7ff555c

Commits on Jul 8, 2026

refactor: replace _total_salary_ sentinel with legacyTotalCompensation field

nesanders
and
claude
committed
5c6bcb1
test: add writer unit tests and fix types.ts formatting

nesanders
and
claude
committed
e9bbdcd

Commits on Jul 10, 2026

fix: track reparse progress via GCS metadata instead of Firestore array

nesanders
and
claude
committed
0e03c10
fix: get year from Firestore registrant doc instead of parsing URL

nesanders
and
claude
committed
c8ba40d
chore: ignore Python pycache/bytecode and remove tracked .pyc files

nesanders
and
claude
committed
3f83938
Redesign the Learn section around a hub page
k-g-k
committed
f144667
Reword the How section intro and Testify in writing step
k-g-k
committed
4c96e4f
Let scroll drive the legislative process accordion
k-g-k
committed
658cf43
Animate the process accordion; revert scroll-driven opening
k-g-k
committed
369baa0
Land a clicked chapter below the rail without a corrective jump
k-g-k
committed
6fad899
Reword the Individuals and Organizations taglines
k-g-k
committed
bb16018
Rework the process page: the rail follows the reader
k-g-k
committed
b6b9fd3
Reduce the rail's look-ahead to 120px
k-g-k
committed
8415911
Make the rail behave differently going up than going down
k-g-k
committed
0255162
Trigger Vercel preview
k-g-k
committed
59ab45a
Land every rail-click chapter cleanly below the sticky nav
k-g-k
committed
9a3b1a5
Fix two mobile navbar/rail issues
k-g-k
committed
1144bd7
Keep the persona tabs on one row with short labels
k-g-k
committed
9743156
Show the active-state look on persona-tab hover, instantly
k-g-k
committed
cdb4bfb
Make the active mobile nav dropdown item bold white
k-g-k
committed
96880c4
Stop the rail's initial-scroll jank on mobile
k-g-k
committed
cd92a1f
Loosen line-height and padding on the AI Research Tools page
k-g-k
committed
d885c2e
Hold rail highlight tracking until it locks, to smooth mobile scroll
k-g-k
committed
c20717e
Reset the legislative-process page to the top on load
k-g-k
committed
6859a8e
Attach the rail's scroll tracking only once it pins (mobile)
k-g-k
committed
50d9c0a
Reset the rail highlight to the first stage when unpinned at the top
k-g-k
committed
f778e7a

Commits on Jul 11, 2026

Merge pull request #2158 from nesanders/lobbying-data-pipeline
Mephistic
authored
8653f0f

Commits on Jul 13, 2026

Deep-link the retired Learn pages to the sections that replaced them
k-g-k
committed
2a9b500

Commits on Jul 14, 2026

update README.md
allcontributors[bot]
authored
6438e58
update .all-contributorsrc
allcontributors[bot]
authored
b908101
update README.md
allcontributors[bot]
authored
3bb53f2
update .all-contributorsrc
allcontributors[bot]
authored
aa5fd0d
Merge pull request #2184 from codeforboston/all-contributors/add-J-C-L
mertbagt
authored
e65a461
Merge branch 'main' into all-contributors/add-Smoss
mertbagt
authored
d162bd6
Merge pull request #2185 from codeforboston/all-contributors/add-Smoss
mertbagt
authored
d8505e1
update README.md
allcontributors[bot]
authored
5d9ed90
update .all-contributorsrc
allcontributors[bot]
authored
a0be179
Merge pull request #2186 from codeforboston/all-contributors/add-c-tonneslan
mertbagt
authored
d0db118
update README.md
allcontributors[bot]
authored
9d3be1c
update .all-contributorsrc
allcontributors[bot]
authored
f4713f0
Merge pull request #2187 from codeforboston/all-contributors/add-VenkateshDevarakonda0706
mertbagt
authored
5eda59a
update README.md
allcontributors[bot]
authored
e5b68ad
update .all-contributorsrc
allcontributors[bot]
authored
41309b3
Merge pull request #2188 from codeforboston/all-contributors/add-ACoullard
mertbagt
authored
ef1ee2f
update README.md
allcontributors[bot]
authored
4f81449
update .all-contributorsrc
allcontributors[bot]
authored
73905b5
Merge pull request #2189 from codeforboston/all-contributors/add-LexaMichaelides
mertbagt
authored
13699d1
update README.md
allcontributors[bot]
authored
4506409
update .all-contributorsrc
allcontributors[bot]
authored
d5977b5
Merge pull request #2190 from codeforboston/all-contributors/add-fastfadingviolets
mertbagt
authored
96a368e
update README.md
allcontributors[bot]
authored
4075242
update .all-contributorsrc
allcontributors[bot]
authored
a2a302c
Merge pull request #2191 from codeforboston/all-contributors/add-AstorDG
mertbagt
authored
f40dcbd
update README.md
allcontributors[bot]
authored
a0c213f
update .all-contributorsrc
allcontributors[bot]
authored
1ae1496
Merge pull request #2192 from codeforboston/all-contributors/add-n1b11
mertbagt
authored
dbfee3b
update README.md
allcontributors[bot]
authored
ae77235
update .all-contributorsrc
allcontributors[bot]
authored
75554ec
Merge pull request #2193 from codeforboston/all-contributors/add-nssensalo
mertbagt
authored
02b2244
update README.md
allcontributors[bot]
authored
28e9f71
update .all-contributorsrc
allcontributors[bot]
authored
5173422
Merge pull request #2194 from codeforboston/all-contributors/add-jasrose
mertbagt
authored
21f1985
Replace the icons on the about testimony page's why section to be more representative of their semantic meaning. Icons have different sizes to correct for a visual perception illusion. Address late…
k-g-k
committed
a486e5e
Add a Writing Effective Testimony page and restyle the Learn hub
k-g-k
committed
609f566
Merge remote-tracking branch 'upstream/main' into learn-hub-redesign
k-g-k
committed
bc734d2
Guard the Learn icon lists, fix the mobile Learn dropdown, and edit copy
k-g-k
committed
086f96b
Fix the mobile Learn dropdown hover and active-row states
k-g-k
committed
22585bd

Commits on Jul 15, 2026

Add Firestore indexes for lobbying explorer (#2199)

nesanders
and
claude
authored
02e0a49
Multi video backend (#2153)
delexagon
authored
99b84fa

Commits on Jul 17, 2026

Commits on Jul 18, 2026

Improve About-page accessibility without visual changes
k-g-k
committed
38abba5
Refine About/Learn spacing and fix mobile Our Team issues
k-g-k
committed
be235d2

Commits on Jul 19, 2026

Merge pull request #2196 from k-g-k/learn-hub-redesign
mertbagt
authored
c4c60a1

Commits on Jul 20, 2026

update README.md
allcontributors[bot]
authored
834ca76
update .all-contributorsrc
allcontributors[bot]
authored
a6b0aad
Merge pull request #2202 from codeforboston/all-contributors/add-k-g-k
mertbagt
authored
032a9b2

Commits on Jul 21, 2026

fix: replace homepage screenshot with Bills search preview (#2206)

nesanders
and
claude
authored
34cd7d9
[Our Team page, Partners tab] remove shoopty whoop (#2201)
mertbagt
authored
6aa83e2
fix org socials (#2197)
mertbagt
authored
c4148f8

Commits on Jul 22, 2026

[Legislator Page] pipe in Maple profile (#2181)
mertbagt
authored
7dfb973
Add OCPF campaign finance scraper and Finance tab (#2195)
J-C-L
authored
91d239d
Election scraping (#2178)
delexagon
authored
eee1444

Commits on Jul 25, 2026

Revise Learn and About page copy and fix typos (#2207)
k-g-k
authored
24ec2d0

Commits on Jul 29, 2026

Pass through bucketName (#2208)
delexagon
authored
2919c33
Finance scrapper (#2209)
J-C-L
authored
f5c8005
Election bugfix
delexagon
committed
c46d5c1

Commits on Jul 30, 2026

Link legislator tabs to URL hashes
ericpastorm
committed
c571307

Commits on Aug 4, 2026

Merge pull request #2210 from delexagon/election-bugfix
mertbagt
authored
7acc3ae

Commits on Aug 5, 2026

Commits on Aug 18, 2026

Fix vote modal (#2217)
delexagon
authored
1e037fb

Commits on Aug 20, 2026

Commits on Aug 21, 2026

update README.md
allcontributors[bot]
authored
e373cec
update .all-contributorsrc
allcontributors[bot]
authored
2a9df5a
Merge pull request #2220 from codeforboston/all-contributors/add-slarson
mertbagt
authored
9a2a9ba

Commits on Aug 23, 2026

fix(lobbying): weekly-scraper cursor exceeds Firestore's 1MB document limit

nesanders
and
claude
committed
1ced6c5