fix: keep leading zeros in every code column - #420
Merged
Conversation
Extends DOI-USGS#419 from the two Samples getters to every delimited response the package parses, and applies one rule to all of them. A code column is named by a ``code`` suffix, the RDB abbreviation ``_cd``, or a code vocabulary in the name (``identifier``, ``huc``, ``fips``); a name ending in ``count`` is a tally and keeps numeric inference. ``dataretrieval._csv`` owns that rule, exposing ``read_code_csv`` for a CSV body and ``code_columns`` for a caller that parses the body itself. Fixed as a result: - ``waterdata.get_samples`` returned ``USGSpcode`` 00060 as 60 and HUC12 070700050502 as 70700050502 (the DOI-USGS#419 fix, unchanged in effect). - ``nwis.get_info`` / ``what_sites`` / ``get_record(service="site")`` returned ``huc_cd`` 02060005 as 2060005, and dropped the zeros from ``state_cd``, ``county_cd`` and ``district_cd``. ``rdb.read_rdb`` now detects code columns from the RDB header rather than the caller enumerating them, so ``_NWIS_RDB_DTYPES`` holds only the names the rule cannot infer. - WQP's ``AlternateLocation_IdentifierCount`` was read as text because "Identifier" appears in its name, giving one WQX3 column two dtypes depending on which adapter fetched it. ``nwdc`` drops its hard-coded ``{huc12_id: str}`` for the shared reader, which covers that column by name. The OGC getters (``waterdata``, ``ngwmn``), NWIS JSON, NLDI and StreamStats were audited and need no change: their codes arrive as JSON strings and numeric coercion there is a fixed list of measurement columns. tests/data/samples_results.txt and tests/data/wqp3_results.txt were recorded through pandas and had already lost their zeros (pcode 61, HUC8 7090002), so neither could fail on this bug. Both are re-recorded from their live queries and now assert the padded values. Also: the shared reader's tests move to tests/_csv_test.py, WQP's parser test becomes a getter-level regression, and NEWS records the behavior change per service plus the same correction shipped undocumented for WQP in 1.2.0.
thodson-usgs
force-pushed
the
preserve-code-columns
branch
from
September 9, 2026 20:07
fdc24ec to
88125dd
Compare
thodson-usgs
marked this pull request as ready for review
September 9, 2026 20:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #419, merged as c2b2507. Closes #411.
@tevinch diagnosed the Samples dtype loss, wrote the shared two-pass reader,
and moved WQP's parser into it. This PR extends that work to the rest of the
package after an audit turned up three more instances of the same defect, and
folds the two NEWS entries into one.
The bug
A bare
pandas.read_csvinfers a zero-padded code as a number, so a parametercode
00060arrives as60and HUC807090002as7090002. Nothing raises;a caller sees a plausible number.
WQP was fixed in #311 (05/31/2026) with a private
_read_wqp_csvhelper.Because the helper stayed private to that adapter, the pattern was free to
reappear, and it had already reappeared: the Samples getter was written the
same way in March 2025, and every later refactor carried the line forward.
#419 relocated the helper into
dataretrieval._csv, which is what makes therest of this a small change.
What this PR does
One rule decides what a code column is — a
codesuffix, the RDBabbreviation
_cd, oridentifier/huc/fipsin the name, with acountsuffix excluded — and every delimited response is parsed through it.dataretrieval._csvowns it:read_code_csvfor a CSV body,code_columnsfor a caller that parses the body itself.
Fixed as a result:
get_info(),what_sites(), andget_record(service="site")returned
huc_cd02060005as2060005, and dropped the zeros fromstate_cd,county_cd(011→11), anddistrict_cd.rdb.read_rdbnow detects code columns from the RDB header instead of the caller
enumerating them, so
_NWIS_RDB_DTYPESholds only the names the rule cannotinfer.
AlternateLocation_IdentifierCountwas read as text because"Identifier" appears in its name, so one WQX3 column had two dtypes
depending on which adapter fetched it — and a sibling count,
ResultBiological_GroupSummaryCount, was numeric in the same frame. Thecountrule settles all three, and Preserve Samples CSV identifiers through shared parsing #419'sinfer_columnsparameter and itsmagic string at the Samples call site are no longer needed.
{huc12_id: str}for the shared reader, whichcovers that column by name.
Also from reviewing #419:
wqpimportsread_code_csvunder its own namerather than aliasing it back to
_read_wqp_csv;_csvmoves to thedependency-free floor of the layers contract, whose stated criterion it meets;
the shared reader gains its own component-layer tests in
tests/_csv_test.pywhile WQP's regression becomes a getter-level test.
Audited and unaffected
The
waterdataOGC getters,ngwmn, NWIS JSON, NLDI, StreamStats, andwaterdata.get_codes()need no change. Their codes arrive as JSON strings, andnumeric coercion in the OGC path is a fixed allow-list of measurement columns
(
altitude,drainage_area,value, …) with no code column in it. Verifiedagainst the recorded fixtures:
00060,09,025, and070900020504allsurvive that path.
Fixtures
tests/data/samples_results.txtandtests/data/wqp3_results.txtwererecorded through pandas and had already lost their zeros —
USGSpcode61,HUC8
7090002, county25. Neither could fail on the bug it was meant toguard, which is why WQP's regression coverage after #311 rested entirely on a
synthetic CSV. Both are re-recorded from their own live queries and now assert
the padded values. Samples grew 67 → 79 rows (real data added since 2025) and
WQX3 gained a column, so two shape assertions move to
(79, 187)and(5, 187).Behavior change
Documented in
NEWS.mdper service, with the remedy: compare against thepadded string (
== "00060") or call.astype(int)where the number is whatyou want. A comparison against a number, or a merge onto a numeric key, now
matches nothing instead of raising.
NEWS.mdalso carries a correction: #311 changed the dtypes of ninewqpgetters and shipped in 1.2.0 with no entry at all. That release is not being
changed — the note records what already happened, so the history is complete
for the next release.
Testing
Local, on Python 3.12 / pandas 3.0.5:
coverage run -m pytest tests/— 1163 passed, 12 live deselected; coverage98.93% against the 98.9 ratchet.
ruff check,ruff format --check,mypy,lint-imports(8 contractskept),
xenon,complexipy, and everypre-commithook overmain...HEAD.huc_cdcomes back as2060005, and the Samples codes as bare integers.(
00061,04100005) and that NWIS expanded site output padscounty_cdto011— the fixtures, not the services, were wrong.Not run here: the live-marked suite (
pytest -m live) beyond the spot checksabove, and the docs build.