Skip to content

fix: keep leading zeros in every code column - #420

Merged
thodson-usgs merged 1 commit into
DOI-USGS:mainfrom
thodson-usgs:preserve-code-columns
Sep 9, 2026
Merged

fix: keep leading zeros in every code column#420
thodson-usgs merged 1 commit into
DOI-USGS:mainfrom
thodson-usgs:preserve-code-columns

Conversation

@thodson-usgs

@thodson-usgs thodson-usgs commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Builds on #419, merged as c2b2507. Closes #411.

@tevinch diagnosed the Samples dtype loss, wrote the shared two-pass reader,
and moved WQP's parser into it. This PR extends that work to the rest of the
package after an audit turned up three more instances of the same defect, and
folds the two NEWS entries into one.

The bug

A bare pandas.read_csv infers a zero-padded code as a number, so a parameter
code 00060 arrives as 60 and HUC8 07090002 as 7090002. Nothing raises;
a caller sees a plausible number.

WQP was fixed in #311 (05/31/2026) with a private _read_wqp_csv helper.
Because the helper stayed private to that adapter, the pattern was free to
reappear, and it had already reappeared: the Samples getter was written the
same way in March 2025, and every later refactor carried the line forward.
#419 relocated the helper into dataretrieval._csv, which is what makes the
rest of this a small change.

What this PR does

One rule decides what a code column is — a code suffix, the RDB
abbreviation _cd, or identifier / huc / fips in the name, with a
count suffix excluded — and every delimited response is parsed through it.
dataretrieval._csv owns it: read_code_csv for a CSV body, code_columns
for a caller that parses the body itself.

Fixed as a result:

  • NWIS: get_info(), what_sites(), and get_record(service="site")
    returned huc_cd 02060005 as 2060005, and dropped the zeros from
    state_cd, county_cd (01111), and district_cd. rdb.read_rdb
    now detects code columns from the RDB header instead of the caller
    enumerating them, so _NWIS_RDB_DTYPES holds only the names the rule cannot
    infer.
  • WQP: AlternateLocation_IdentifierCount was read as text because
    "Identifier" appears in its name, so one WQX3 column had two dtypes
    depending on which adapter fetched it — and a sibling count,
    ResultBiological_GroupSummaryCount, was numeric in the same frame. The
    count rule settles all three, and Preserve Samples CSV identifiers through shared parsing #419's infer_columns parameter and its
    magic string at the Samples call site are no longer needed.
  • NWDC drops its hard-coded {huc12_id: str} for the shared reader, which
    covers that column by name.

Also from reviewing #419: wqp imports read_code_csv under its own name
rather than aliasing it back to _read_wqp_csv; _csv moves to the
dependency-free floor of the layers contract, whose stated criterion it meets;
the shared reader gains its own component-layer tests in tests/_csv_test.py
while WQP's regression becomes a getter-level test.

Audited and unaffected

The waterdata OGC getters, ngwmn, NWIS JSON, NLDI, StreamStats, and
waterdata.get_codes() need no change. Their codes arrive as JSON strings, and
numeric coercion in the OGC path is a fixed allow-list of measurement columns
(altitude, drainage_area, value, …) with no code column in it. Verified
against the recorded fixtures: 00060, 09, 025, and 070900020504 all
survive that path.

Fixtures

tests/data/samples_results.txt and tests/data/wqp3_results.txt were
recorded through pandas and had already lost their zeros — USGSpcode 61,
HUC8 7090002, county 25. Neither could fail on the bug it was meant to
guard, which is why WQP's regression coverage after #311 rested entirely on a
synthetic CSV. Both are re-recorded from their own live queries and now assert
the padded values. Samples grew 67 → 79 rows (real data added since 2025) and
WQX3 gained a column, so two shape assertions move to (79, 187) and
(5, 187).

Behavior change

Documented in NEWS.md per service, with the remedy: compare against the
padded string (== "00060") or call .astype(int) where the number is what
you want. A comparison against a number, or a merge onto a numeric key, now
matches nothing instead of raising.

NEWS.md also carries a correction: #311 changed the dtypes of nine wqp
getters and shipped in 1.2.0 with no entry at all. That release is not being
changed — the note records what already happened, so the history is complete
for the next release.

Testing

Local, on Python 3.12 / pandas 3.0.5:

  • coverage run -m pytest tests/ — 1163 passed, 12 live deselected; coverage
    98.93% against the 98.9 ratchet.
  • ruff check, ruff format --check, mypy, lint-imports (8 contracts
    kept), xenon, complexipy, and every pre-commit hook over
    main...HEAD.
  • The two new regressions fail on the pre-Preserve Samples CSV identifiers through shared parsing #419 tree: huc_cd comes back as
    2060005, and the Samples codes as bare integers.
  • Live checks against the real services confirmed the wire format pads
    (00061, 04100005) and that NWIS expanded site output pads county_cd to
    011 — the fixtures, not the services, were wrong.

Not run here: the live-marked suite (pytest -m live) beyond the spot checks
above, and the docs build.

Extends DOI-USGS#419 from the two Samples getters to every delimited response the
package parses, and applies one rule to all of them.

A code column is named by a ``code`` suffix, the RDB abbreviation ``_cd``, or a
code vocabulary in the name (``identifier``, ``huc``, ``fips``); a name ending
in ``count`` is a tally and keeps numeric inference. ``dataretrieval._csv``
owns that rule, exposing ``read_code_csv`` for a CSV body and ``code_columns``
for a caller that parses the body itself.

Fixed as a result:

- ``waterdata.get_samples`` returned ``USGSpcode`` 00060 as 60 and HUC12
  070700050502 as 70700050502 (the DOI-USGS#419 fix, unchanged in effect).
- ``nwis.get_info`` / ``what_sites`` / ``get_record(service="site")`` returned
  ``huc_cd`` 02060005 as 2060005, and dropped the zeros from ``state_cd``,
  ``county_cd`` and ``district_cd``. ``rdb.read_rdb`` now detects code columns
  from the RDB header rather than the caller enumerating them, so
  ``_NWIS_RDB_DTYPES`` holds only the names the rule cannot infer.
- WQP's ``AlternateLocation_IdentifierCount`` was read as text because
  "Identifier" appears in its name, giving one WQX3 column two dtypes
  depending on which adapter fetched it.

``nwdc`` drops its hard-coded ``{huc12_id: str}`` for the shared reader, which
covers that column by name. The OGC getters (``waterdata``, ``ngwmn``), NWIS
JSON, NLDI and StreamStats were audited and need no change: their codes arrive
as JSON strings and numeric coercion there is a fixed list of measurement
columns.

tests/data/samples_results.txt and tests/data/wqp3_results.txt were recorded
through pandas and had already lost their zeros (pcode 61, HUC8 7090002), so
neither could fail on this bug. Both are re-recorded from their live queries
and now assert the padded values.

Also: the shared reader's tests move to tests/_csv_test.py, WQP's parser test
becomes a getter-level regression, and NEWS records the behavior change per
service plus the same correction shipped undocumented for WQP in 1.2.0.
@thodson-usgs
thodson-usgs marked this pull request as ready for review September 9, 2026 20:13
@thodson-usgs
thodson-usgs merged commit 0d345a7 into DOI-USGS:main Sep 9, 2026
11 checks passed
@thodson-usgs
thodson-usgs deleted the preserve-code-columns branch September 9, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve Samples identifier columns as text

1 participant