Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ can predict where a thing is defined.
- `dataretrieval/transport/` — service-neutral request code (HTTP, retry,
pagination, fan-out). It names no service and no protocol, and is not public API.
- Leading-underscore top-level modules are private; the dependency-free *leaves*
are at the bottom of the stack so anything may use them without pulling in the
are at the bottom of the stack so anything may use them without importing the
rest of the package. Check for an existing leaf before writing a small helper.
- **`.importlinter` records where every module belongs.** Its `layers` contract
lists every top-level module in dependency order and is `exhaustive = True`,
Expand All @@ -62,7 +62,7 @@ can predict where a thing is defined.
`*_test.ipynb` at the top level are untracked local scratch — don't edit,
commit, or cite them.
- Exclude `.claude/worktrees/` from searches and edits; stale worktrees there
pollute results.
add spurious results.

## Environment
- `pip install .[test,nldi]` (CI uses pip, not uv, despite `uv.lock`).
Expand All @@ -77,8 +77,8 @@ can predict where a thing is defined.
a merge gate: branch coverage with a `fail_under` ratchet in
`[tool.coverage.report]`. Cover the uncovered *branch*, not the number -- a
test written only to mark a line as covered catches nothing and adds
maintenance. If a path is genuinely unreachable, add it to `exclude_also`
with a reason, or leave the ratchet alone.
maintenance. If a path is unreachable, add it to `exclude_also`
with a reason, or leave the ratchet unchanged.
- Types: `mypy` (`strict = true` in `pyproject.toml`; CI runs it over the
PR-merged-into-main, so bare `dict`/`list` annotations fail there even if they
pass on your branch).
Expand All @@ -87,7 +87,7 @@ can predict where a thing is defined.
- Docs: install docs deps, `ipython kernel install --name "python3" --user`, then
`make html` from `docs/`. `make docs` adds doctest+linkcheck (network-dependent).

## Testing gotchas
## Testing notes
- The suite is offline by default: `addopts = "-m 'not live'"`. Tests marked
`@pytest.mark.live` call real USGS services and run on a schedule
(`.github/workflows/live-api.yml`); run them locally with `pytest tests/ -m live`.
Expand Down Expand Up @@ -125,17 +125,17 @@ raise states the problem and then the action that fixes it, in that order.
a real parameter of the function the *caller* called — not a private helper's
local, not a prose label — and following it literally must produce a working
call. Messages that read well have failed all three: `datetime_input` was a
private local no getter accepts, `configure(Configuration(...))` was a silent
no-op because `configure` is a context manager, `pip install
private local no getter accepts, `configure(Configuration(...))` was a no-op that raised
nothing because `configure` is a context manager, `pip install
dataretrieval[nldi]` globs in zsh, and a navigation missing its `data_source`
wrote `None` into the URL and returned an empty frame. Run the corrected
call against the real service; wording review does not catch these.
- Shared checks take the caller's argument name. `_validate_data_source`,
`_format_api_dates`, and `require_one_of` all accept a `name=` so the subject
of the message is the argument that was actually passed. A helper that hard-codes
of the message is the argument that was passed. A helper that hard-codes
one noun reports the wrong parameter the moment a second call site reuses it.
- Prefer raising over returning something empty when the library cannot tell
"no data" from "the service misbehaved": a caller that gets an empty frame has
- Prefer raising over returning something empty when the library cannot distinguish
an empty result from a service failure: a caller that gets an empty frame has
no signal to act on. `nldi._query_nldi` is the deliberate exception — a 200
with a non-JSON body becomes an empty GeoDataFrame by design.

Expand All @@ -144,7 +144,7 @@ raise states the problem and then the action that fixes it, in that order.
`httpx` and tests mock with `httpx_mock`.
- Public getters return `(DataFrame, metadata)`.
- `dataretrieval/__init__.py` imports the service modules by name and lists them
in `__all__`; it does not star-import them, so a getter is reached through its
in `__all__`; it does not star-import them, so a getter is accessed through its
module (`dataretrieval.nwis.get_record`), never from the top level. `nldi` is
deliberately absent — it needs `geopandas` at import time, so it is imported on
demand. `dataretrieval/waterdata/__init__.py` controls Water Data exports via
Expand Down
19 changes: 10 additions & 9 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ conversation use them the same way. Architectural decisions are recorded in
Two kinds of term are defined here, and they impose different obligations
(ADR 0013).

**Core terms** are ours. The package invented them and no service has a claim on
**Core terms** are ours. The package invented them and no service defines
them — *chunk*, *page*, *fan-out*, *source*, *dialect*, *leaf*. One spelling,
everywhere it appears: prose, identifiers, tests. Where a core term conflicts
with a name in the code the term is authoritative and the name is legacy,
Expand Down Expand Up @@ -47,7 +47,7 @@ location per request — or because the caller asked for it. Both produce chunks
the reason is not part of the term.

**Plan** — An enumeration of a query's chunks: how many there are, and what each
one is. A plan says how a query divides; it does not execute. Computing a plan
one is. A plan describes how a query divides; it does not execute. Computing a plan
is protocol-specific — a byte budget, a per-location rule — while executing one
is not, which is why the two are kept in separate modules.

Expand All @@ -66,7 +66,8 @@ rate limit, a service error, a timeout. Distinguished from a **deterministic
failure**, which would fail identically every time — an unresolvable hostname,
an unsupported scheme, a malformed request. Only transient failures are
retried, and only transient failures produce a resumable interruption. Both
answers follow from one judgement about what a failure means, and must agree.
answers follow from one judgement about what a failure means, and must be
consistent.

**Stall timeout** — How long a call may receive nothing at all before retrying
stops, measured from when data last arrived rather than from the call's start.
Expand Down Expand Up @@ -135,7 +136,7 @@ resemblance is the public contract, not duplication to be removed.

**Monitoring location** — A place where measurements are recorded.

*Domain term.* The services disagree, and each adapter keeps its own service's
*Domain term.* The services differ, and each adapter keeps its own service's
spelling in its parameters: NWIS `site_no` and `sites=`, WQP `Station` and
`siteid`, Water Data `monitoring_location_id`, NGWMN's `sites` collection. Where
a service names a thing `site-types` or `site_type_code`, that is its vocabulary
Expand Down Expand Up @@ -177,8 +178,8 @@ argument on four adapters and resolves through no chain at all; the settings are
the list the configuration system recognizes.

**Scope** — How much of the package a setting's value applies to: the whole
package, or one adapter. Orthogonal to source: the scope says who a value is
for, the source says where it came from, and precedence orders sources first,
package, or one adapter. Orthogonal to source: the scope states which part of the package a value is
for, the source states where it came from, and precedence orders sources first,
scopes within them. ADR 0010's word for a scope level is *tier* — the top-level
tier that remains, the host or gateway tier it defers.

Expand All @@ -192,7 +193,7 @@ rejects a setting it has no use for, rather than accepting and ignoring it.

The scope is the *adapter*, not the service and not the host, because the
adapter is what owns the conventions being tuned. The API key shows where the
boundary falls: it belongs to the gateway fronting a host, so Water Data and
boundary is: it belongs to the gateway fronting a host, so Water Data and
NGWMN — two adapters, one host — necessarily share one key and one quota pool.
Credentials are host-scoped; tunables are adapter-scoped.

Expand All @@ -202,7 +203,7 @@ default. The order is resolved per setting rather than per source: a value
supplied for one setting does not displace another setting's value from a lower
source.

*Core term.* The accepted records already say it: ADR 0009 resolves settings
*Core term.* The accepted records already use it: ADR 0009 resolves settings
by source, and ADR 0010 keeps precedence *source-major*. ADR 0010's *tier* is a
different axis — the scope — and ADR 0011's *rungs* are positions of its merged
precedence ladder, where sources and scopes interleave. Neither is a second
Expand Down Expand Up @@ -247,7 +248,7 @@ for one adapter. Both narrow to a single adapter; only one of them is something
the caller wrote.

All three are called "the default" in casual use, and they are not the same
value. Where the distinction matters — reporting what a call will actually use
value. Where the distinction matters — reporting what a call will use
— say which one is meant.

## Boundaries
Expand Down
26 changes: 13 additions & 13 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ test run neither depends on USGS uptime nor spends anyone's rate limit.

The exception is a small set of tests marked `live`, which query the real
services to notice when an upstream API changes -- something a mock cannot
tell us, because the mock is what would need updating. They are deselected by
detect, because the mock is what would need updating. They are deselected by
default and run on a nightly schedule
([live-api.yml](https://github.com/DOI-USGS/dataretrieval-python/blob/main/.github/workflows/live-api.yml)).
Run them locally with:
Expand All @@ -118,7 +118,7 @@ This package keeps its general mechanisms in dependency-free leaves --
it, and it is the only module that reads the environment for a setting),
`transport.links.resolve_next_url` for pagination cursors. Each of those has
been re-implemented at least once by someone who did not know it was there, and
the copies drift: the same question gets a different cycle guard, a different
the copies drift: the same check gets a different cycle guard, a different
error message, a different edge case. None of the automated checks catch it,
because two eight-line helpers are below the clone detector's minimum size and
neither one couples nor complicates anything. A grep for the mechanism you are
Expand All @@ -139,7 +139,7 @@ lint-imports
```

The last three come from `pip install -e '.[metrics]'`, and each has a pre-commit
hook running the identical check, so a clean pre-commit run means CI agrees.
hook running the identical check, so a clean pre-commit run means CI will pass.

`coverage report` is a ratchet too. The threshold is in
`[tool.coverage.report]` in `pyproject.toml` and is set to the measured value,
Expand All @@ -149,16 +149,16 @@ why in the commit.

Coverage is measured with branches on, because most of what this package gets
wrong is a branch rather than a line -- a dispatch arm routing to the wrong
getter, an error path that never executes, a fallback that quietly becomes the
norm. Cover the *uncovered branch*, not the percentage: a test written only to
getter, an error path that never executes, a fallback that becomes the
norm unnoticed. Cover the *uncovered branch*, not the percentage: a test written only to
mark a line as covered adds maintenance and catches nothing. If a path cannot be
reached without contorting the code, exclude it in
reached without restructuring the code, exclude it in
`[tool.coverage.report] exclude_also` with a reason, or leave the ratchet where
it is. Either costs less than a test that adds maintenance and catches nothing.

The blocking run is a single Linux job. The OS/Python matrix reports its own
number with `--fail-under=0`, because several tests are POSIX-only and a
Windows run genuinely measures a smaller suite.
Windows run measures a smaller suite.

For the same reason, the threshold assumes the whole suite: on Windows, or
without the `nldi` extra installed, some tests skip and the local number comes
Expand All @@ -167,8 +167,8 @@ in under the gate through no fault of your change. Run
ratchet.

`xenon` and `complexipy` are complexity ratchets: the thresholds are the
tightest the package passes today, so they fail only when a change pushes a
score above today's. They disagree because they count different things. `xenon` counts
tightest the package passes today, so they fail only when a change raises a
score above today's. Their scores differ because they count different things. `xenon` counts
branches (cyclomatic complexity), so a large flat dispatch scores high;
`complexipy` counts how hard the control flow is to follow (cognitive
complexity), so it scores that dispatch lower and nesting higher. Both name the
Expand Down Expand Up @@ -211,9 +211,9 @@ Duplication, coupling, cohesion, dependency depth, and dead code are tracked by
[`pyscn`](https://github.com/ludo-technologies/pyscn) on a weekly schedule
([code-health.yml](https://github.com/DOI-USGS/dataretrieval-python/blob/main/.github/workflows/code-health.yml)),
which attaches an HTML and a JSON report to each run. Nothing gates on it. These
measures move over months rather than commits.
measures change over months rather than commits.

You do not need it to contribute. It answers "what should we clean up next?" --
You do not need it to contribute. It identifies what to clean up next --
including for an agent working on this repo, which gets a whole-package
structural overview from one command:

Expand All @@ -224,9 +224,9 @@ pyscn analyze dataretrieval # HTML report, or --json for the numbers
```

Read its findings as suggestions, not conclusions. Its clone detector flags this
package's per-collection getters -- thin, heavily documented wrappers whose
package's per-collection getters -- thin, extensively documented wrappers whose
bodies are necessarily similar -- and collapsing them into one parameterized
function would sacrifice the documented public surface for a metric. Its
function would give up the documented public surface for a metric. Its
dependency-injection heuristics expect a class-oriented design this package
deliberately does not have.

Expand Down
Loading