diff --git a/AGENTS.md b/AGENTS.md index 29b443a92..d7d71f104 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,7 +35,7 @@ can predict where a thing is defined. - `dataretrieval/transport/` — service-neutral request code (HTTP, retry, pagination, fan-out). It names no service and no protocol, and is not public API. - Leading-underscore top-level modules are private; the dependency-free *leaves* - are at the bottom of the stack so anything may use them without pulling in the + are at the bottom of the stack so anything may use them without importing the rest of the package. Check for an existing leaf before writing a small helper. - **`.importlinter` records where every module belongs.** Its `layers` contract lists every top-level module in dependency order and is `exhaustive = True`, @@ -62,7 +62,7 @@ can predict where a thing is defined. `*_test.ipynb` at the top level are untracked local scratch — don't edit, commit, or cite them. - Exclude `.claude/worktrees/` from searches and edits; stale worktrees there - pollute results. + add spurious results. ## Environment - `pip install .[test,nldi]` (CI uses pip, not uv, despite `uv.lock`). @@ -77,8 +77,8 @@ can predict where a thing is defined. a merge gate: branch coverage with a `fail_under` ratchet in `[tool.coverage.report]`. Cover the uncovered *branch*, not the number -- a test written only to mark a line as covered catches nothing and adds - maintenance. If a path is genuinely unreachable, add it to `exclude_also` - with a reason, or leave the ratchet alone. + maintenance. If a path is unreachable, add it to `exclude_also` + with a reason, or leave the ratchet unchanged. - Types: `mypy` (`strict = true` in `pyproject.toml`; CI runs it over the PR-merged-into-main, so bare `dict`/`list` annotations fail there even if they pass on your branch). @@ -87,7 +87,7 @@ can predict where a thing is defined. - Docs: install docs deps, `ipython kernel install --name "python3" --user`, then `make html` from `docs/`. `make docs` adds doctest+linkcheck (network-dependent). -## Testing gotchas +## Testing notes - The suite is offline by default: `addopts = "-m 'not live'"`. Tests marked `@pytest.mark.live` call real USGS services and run on a schedule (`.github/workflows/live-api.yml`); run them locally with `pytest tests/ -m live`. @@ -125,17 +125,17 @@ raise states the problem and then the action that fixes it, in that order. a real parameter of the function the *caller* called — not a private helper's local, not a prose label — and following it literally must produce a working call. Messages that read well have failed all three: `datetime_input` was a - private local no getter accepts, `configure(Configuration(...))` was a silent - no-op because `configure` is a context manager, `pip install + private local no getter accepts, `configure(Configuration(...))` was a no-op that raised + nothing because `configure` is a context manager, `pip install dataretrieval[nldi]` globs in zsh, and a navigation missing its `data_source` wrote `None` into the URL and returned an empty frame. Run the corrected call against the real service; wording review does not catch these. - Shared checks take the caller's argument name. `_validate_data_source`, `_format_api_dates`, and `require_one_of` all accept a `name=` so the subject - of the message is the argument that was actually passed. A helper that hard-codes + of the message is the argument that was passed. A helper that hard-codes one noun reports the wrong parameter the moment a second call site reuses it. -- Prefer raising over returning something empty when the library cannot tell - "no data" from "the service misbehaved": a caller that gets an empty frame has +- Prefer raising over returning something empty when the library cannot distinguish + an empty result from a service failure: a caller that gets an empty frame has no signal to act on. `nldi._query_nldi` is the deliberate exception — a 200 with a non-JSON body becomes an empty GeoDataFrame by design. @@ -144,7 +144,7 @@ raise states the problem and then the action that fixes it, in that order. `httpx` and tests mock with `httpx_mock`. - Public getters return `(DataFrame, metadata)`. - `dataretrieval/__init__.py` imports the service modules by name and lists them - in `__all__`; it does not star-import them, so a getter is reached through its + in `__all__`; it does not star-import them, so a getter is accessed through its module (`dataretrieval.nwis.get_record`), never from the top level. `nldi` is deliberately absent — it needs `geopandas` at import time, so it is imported on demand. `dataretrieval/waterdata/__init__.py` controls Water Data exports via diff --git a/CONTEXT.md b/CONTEXT.md index a443d88b4..f76a9f7ee 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -8,7 +8,7 @@ conversation use them the same way. Architectural decisions are recorded in Two kinds of term are defined here, and they impose different obligations (ADR 0013). -**Core terms** are ours. The package invented them and no service has a claim on +**Core terms** are ours. The package invented them and no service defines them — *chunk*, *page*, *fan-out*, *source*, *dialect*, *leaf*. One spelling, everywhere it appears: prose, identifiers, tests. Where a core term conflicts with a name in the code the term is authoritative and the name is legacy, @@ -47,7 +47,7 @@ location per request — or because the caller asked for it. Both produce chunks the reason is not part of the term. **Plan** — An enumeration of a query's chunks: how many there are, and what each -one is. A plan says how a query divides; it does not execute. Computing a plan +one is. A plan describes how a query divides; it does not execute. Computing a plan is protocol-specific — a byte budget, a per-location rule — while executing one is not, which is why the two are kept in separate modules. @@ -66,7 +66,8 @@ rate limit, a service error, a timeout. Distinguished from a **deterministic failure**, which would fail identically every time — an unresolvable hostname, an unsupported scheme, a malformed request. Only transient failures are retried, and only transient failures produce a resumable interruption. Both -answers follow from one judgement about what a failure means, and must agree. +answers follow from one judgement about what a failure means, and must be +consistent. **Stall timeout** — How long a call may receive nothing at all before retrying stops, measured from when data last arrived rather than from the call's start. @@ -135,7 +136,7 @@ resemblance is the public contract, not duplication to be removed. **Monitoring location** — A place where measurements are recorded. -*Domain term.* The services disagree, and each adapter keeps its own service's +*Domain term.* The services differ, and each adapter keeps its own service's spelling in its parameters: NWIS `site_no` and `sites=`, WQP `Station` and `siteid`, Water Data `monitoring_location_id`, NGWMN's `sites` collection. Where a service names a thing `site-types` or `site_type_code`, that is its vocabulary @@ -177,8 +178,8 @@ argument on four adapters and resolves through no chain at all; the settings are the list the configuration system recognizes. **Scope** — How much of the package a setting's value applies to: the whole -package, or one adapter. Orthogonal to source: the scope says who a value is -for, the source says where it came from, and precedence orders sources first, +package, or one adapter. Orthogonal to source: the scope states which part of the package a value is +for, the source states where it came from, and precedence orders sources first, scopes within them. ADR 0010's word for a scope level is *tier* — the top-level tier that remains, the host or gateway tier it defers. @@ -192,7 +193,7 @@ rejects a setting it has no use for, rather than accepting and ignoring it. The scope is the *adapter*, not the service and not the host, because the adapter is what owns the conventions being tuned. The API key shows where the -boundary falls: it belongs to the gateway fronting a host, so Water Data and +boundary is: it belongs to the gateway fronting a host, so Water Data and NGWMN — two adapters, one host — necessarily share one key and one quota pool. Credentials are host-scoped; tunables are adapter-scoped. @@ -202,7 +203,7 @@ default. The order is resolved per setting rather than per source: a value supplied for one setting does not displace another setting's value from a lower source. -*Core term.* The accepted records already say it: ADR 0009 resolves settings +*Core term.* The accepted records already use it: ADR 0009 resolves settings by source, and ADR 0010 keeps precedence *source-major*. ADR 0010's *tier* is a different axis — the scope — and ADR 0011's *rungs* are positions of its merged precedence ladder, where sources and scopes interleave. Neither is a second @@ -247,7 +248,7 @@ for one adapter. Both narrow to a single adapter; only one of them is something the caller wrote. All three are called "the default" in casual use, and they are not the same -value. Where the distinction matters — reporting what a call will actually use +value. Where the distinction matters — reporting what a call will use — say which one is meant. ## Boundaries diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c886cb1c6..015c2b942 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,7 +97,7 @@ test run neither depends on USGS uptime nor spends anyone's rate limit. The exception is a small set of tests marked `live`, which query the real services to notice when an upstream API changes -- something a mock cannot -tell us, because the mock is what would need updating. They are deselected by +detect, because the mock is what would need updating. They are deselected by default and run on a nightly schedule ([live-api.yml](https://github.com/DOI-USGS/dataretrieval-python/blob/main/.github/workflows/live-api.yml)). Run them locally with: @@ -118,7 +118,7 @@ This package keeps its general mechanisms in dependency-free leaves -- it, and it is the only module that reads the environment for a setting), `transport.links.resolve_next_url` for pagination cursors. Each of those has been re-implemented at least once by someone who did not know it was there, and -the copies drift: the same question gets a different cycle guard, a different +the copies drift: the same check gets a different cycle guard, a different error message, a different edge case. None of the automated checks catch it, because two eight-line helpers are below the clone detector's minimum size and neither one couples nor complicates anything. A grep for the mechanism you are @@ -139,7 +139,7 @@ lint-imports ``` The last three come from `pip install -e '.[metrics]'`, and each has a pre-commit -hook running the identical check, so a clean pre-commit run means CI agrees. +hook running the identical check, so a clean pre-commit run means CI will pass. `coverage report` is a ratchet too. The threshold is in `[tool.coverage.report]` in `pyproject.toml` and is set to the measured value, @@ -149,16 +149,16 @@ why in the commit. Coverage is measured with branches on, because most of what this package gets wrong is a branch rather than a line -- a dispatch arm routing to the wrong -getter, an error path that never executes, a fallback that quietly becomes the -norm. Cover the *uncovered branch*, not the percentage: a test written only to +getter, an error path that never executes, a fallback that becomes the +norm unnoticed. Cover the *uncovered branch*, not the percentage: a test written only to mark a line as covered adds maintenance and catches nothing. If a path cannot be -reached without contorting the code, exclude it in +reached without restructuring the code, exclude it in `[tool.coverage.report] exclude_also` with a reason, or leave the ratchet where it is. Either costs less than a test that adds maintenance and catches nothing. The blocking run is a single Linux job. The OS/Python matrix reports its own number with `--fail-under=0`, because several tests are POSIX-only and a -Windows run genuinely measures a smaller suite. +Windows run measures a smaller suite. For the same reason, the threshold assumes the whole suite: on Windows, or without the `nldi` extra installed, some tests skip and the local number comes @@ -167,8 +167,8 @@ in under the gate through no fault of your change. Run ratchet. `xenon` and `complexipy` are complexity ratchets: the thresholds are the -tightest the package passes today, so they fail only when a change pushes a -score above today's. They disagree because they count different things. `xenon` counts +tightest the package passes today, so they fail only when a change raises a +score above today's. Their scores differ because they count different things. `xenon` counts branches (cyclomatic complexity), so a large flat dispatch scores high; `complexipy` counts how hard the control flow is to follow (cognitive complexity), so it scores that dispatch lower and nesting higher. Both name the @@ -211,9 +211,9 @@ Duplication, coupling, cohesion, dependency depth, and dead code are tracked by [`pyscn`](https://github.com/ludo-technologies/pyscn) on a weekly schedule ([code-health.yml](https://github.com/DOI-USGS/dataretrieval-python/blob/main/.github/workflows/code-health.yml)), which attaches an HTML and a JSON report to each run. Nothing gates on it. These -measures move over months rather than commits. +measures change over months rather than commits. -You do not need it to contribute. It answers "what should we clean up next?" -- +You do not need it to contribute. It identifies what to clean up next -- including for an agent working on this repo, which gets a whole-package structural overview from one command: @@ -224,9 +224,9 @@ pyscn analyze dataretrieval # HTML report, or --json for the numbers ``` Read its findings as suggestions, not conclusions. Its clone detector flags this -package's per-collection getters -- thin, heavily documented wrappers whose +package's per-collection getters -- thin, extensively documented wrappers whose bodies are necessarily similar -- and collapsing them into one parameterized -function would sacrifice the documented public surface for a metric. Its +function would give up the documented public surface for a metric. Its dependency-injection heuristics expect a class-oriented design this package deliberately does not have. diff --git a/NEWS.md b/NEWS.md index 80f5d6aa1..7ce8bdf3e 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,34 +1,34 @@ **09/01/2026:** **Announcement:** We at USGS Water Data for the Nation want your feedback! Tell us how we're doing by taking our quick [survey](https://usgswaterresources.gov1.qualtrics.com/jfe/form/SV_07gX8G1DeOtVrH8), available through September 2026. -**08/27/2026:** **Bug fix:** `nwis.get_discharge_peaks` and `nwis.get_record(service='peaks')` silently discarded every peak whose date is only partly known. NWIS zero-fills the unknown part of a historical peak's date -- `YYYY-MM-00` when the day is not known, `YYYY-00-00` when the month is not either (the `Bd` and `Bm` `peak_cd` qualifiers) -- and neither parses as a date, so `preformat_peaks_response` coerced both to `NaT` and then dropped the row along with its discharge value. These are real peaks, and disproportionately a site's largest: site 14105700 lost 20 of its 167 peaks, among them an 1859 flood of 847,000 ft3/s, and 06934500 lost its 1844 peak of 700,000 ft3/s. Such a peak is now kept, with `datetime` left as `NaT`. The date is not completed into one NWIS does not have: a `datetime64` column cannot hold a partial date, so any value there would assert a day the record does not claim. **Behavior change:** peaks queries return more rows than before, and `datetime` may now be `NaT` -- a caller selecting on the datetime index will not see those peaks and should filter on `peak_dt` instead. **Behavior change:** `peak_dt` is no longer removed from the returned frame. It is the only column that holds a censored peak's year, since the peaks response has no `water_yr`, and the only dependable way to tell an unknown day from a known one -- `peak_cd` does not always include the qualifier (22 of 24 censored dates across six sites tested). For peaks with a resolved timestamp alongside explicit `year`/`month`/`day` and a `qualifier` field, use `waterdata.get_peaks()`. +**08/27/2026:** **Bug fix:** `nwis.get_discharge_peaks` and `nwis.get_record(service='peaks')` discarded every peak whose date is only partly known. NWIS zero-fills the unknown part of a historical peak's date -- `YYYY-MM-00` when the day is not known, `YYYY-00-00` when the month is not either (the `Bd` and `Bm` `peak_cd` qualifiers) -- and neither parses as a date, so `preformat_peaks_response` coerced both to `NaT` and then dropped the row along with its discharge value. These are real peaks, and disproportionately a site's largest: site 14105700 lost 20 of its 167 peaks, among them an 1859 flood of 847,000 ft3/s, and 06934500 lost its 1844 peak of 700,000 ft3/s. Such a peak is now kept, with `datetime` left as `NaT`. The date is not completed into one NWIS does not have: a `datetime64` column cannot hold a partial date, so any value there would assert a day the record does not contain. **Behavior change:** peaks queries return more rows than before, and `datetime` may now be `NaT` -- a caller selecting on the datetime index will not see those peaks and should filter on `peak_dt` instead. **Behavior change:** `peak_dt` is no longer removed from the returned frame. It is the only column that holds a censored peak's year, since the peaks response has no `water_yr`, and the only dependable way to tell an unknown day from a known one -- `peak_cd` does not always include the qualifier (22 of 24 censored dates across six sites tested). For peaks with a resolved timestamp alongside explicit `year`/`month`/`day` and a `qualifier` field, use `waterdata.get_peaks()`. **08/26/2026:** **Bug fix:** `nwis.format_response(df, service='peaks')` and `nwis.preformat_peaks_response` raised `KeyError('peak_dt')` on an empty peaks response instead of returning an empty frame. Both are public, and every other service already treated an empty result as a legitimate empty frame rather than an error (issue #171); the peaks branch was missed because it reformats the datetime column before the empty-frame check. Callers can now check `df.empty` rather than catching an exception. A *non-empty* frame with no `peak_dt` column is malformed rather than empty, and still raises. **08/25/2026:** Removed `dataretrieval.ogc.retry`, which only re-exported private helpers. Deprecated `dataretrieval.ogc.interruptions`; import exceptions from `dataretrieval` or `dataretrieval.interruptions` instead. The old path will be removed in a future major release, no earlier than 2027-08-25. -**08/20/2026:** The `state` filter now accepts the five US territories. `dataretrieval.codes.states` held the 50 states and DC, so `ngwmn.get_sites(state='Puerto Rico')`, `waterdata.get_monitoring_locations(state_name='Puerto Rico')` via the unified `state`, and `nwdc.get_wateruse(state='PR')` were refused locally -- while all three services have the data (NGWMN returns 36 Puerto Rico monitoring locations, the Water Data monitoring-locations collection returns Puerto Rico sites, and legacy NWIS lists 1,148 stream sites for `stateCd=PR`). American Samoa, Guam, the Northern Mariana Islands, Puerto Rico and the US Virgin Islands are now in both code tables under their real ANSI/FIPS codes, so every encoding resolves: `'Puerto Rico'`, `'PR'`, `'72'` and `'US:72'` all normalize alike. **Behavior change:** a territory that used to raise `ValueError` now produces a request. A value the table genuinely does not hold still fails fast. +**08/20/2026:** The `state` filter now accepts the five US territories. `dataretrieval.codes.states` held the 50 states and DC, so `ngwmn.get_sites(state='Puerto Rico')`, `waterdata.get_monitoring_locations(state_name='Puerto Rico')` via the unified `state`, and `nwdc.get_wateruse(state='PR')` were refused locally -- while all three services have the data (NGWMN returns 36 Puerto Rico monitoring locations, the Water Data monitoring-locations collection returns Puerto Rico sites, and legacy NWIS lists 1,148 stream sites for `stateCd=PR`). American Samoa, Guam, the Northern Mariana Islands, Puerto Rico and the US Virgin Islands are now in both code tables under their real ANSI/FIPS codes, so every encoding resolves: `'Puerto Rico'`, `'PR'`, `'72'` and `'US:72'` all normalize alike. **Behavior change:** a territory that used to raise `ValueError` now produces a request. A value the table does not hold still fails fast. -**08/20/2026:** Argument checks now share one vocabulary, and every rejection explains how to correct the call. `dataretrieval._validation` owns the message shape for each check that recurs across the adapters -- a value outside a closed vocabulary (`require_one_of`), a missing argument (`require_argument`, `require_together`), a query with no filter at all (`require_any_of`), and arguments that conflict (`require_exactly_one`, `reject_together`) -- so a new check cannot invent its own phrasing, which is how `get_reference_table` came to tell callers who passed a bad `collection` that their *code service* was invalid. Each check takes the caller's own name for the parameter and a remedy for the action it cannot derive, and every check raises `ValueError` -- one class for a bad argument value. **Behavior change:** the text of those rejections moves with them -- `"Unrecognized service: 'x'. get_record serves …"` is now `"Invalid service: 'x'. Valid options are: …"`, and the major-filter and bounding-box rejections in `query_waterdata` / `query_waterservices` are rendered in the shared form. **Behavior change:** the deprecated `nwis` query entry points (`query_waterdata`, `query_waterservices`, `get_record`) now respond to a missing major filter, an incomplete bounding box, or an unknown service with `ValueError` rather than their historic `TypeError` -- `TypeError` remains for a genuinely mistyped argument, such as a non-string `sites`. Code catching `TypeError` there, or matching on the old strings, must update. **Bug fix:** a `None` passed as a major filter (`query_waterservices(service='dv', sites=None)`) counted as a filter and reached the service as an empty `sites=`; `None` now means not supplied, and the call is refused with the filters that would have served. **Bug fix:** four messages told callers to do something that raised again or named a parameter their getter does not accept -- `nldi.get_features(comid=…, feature_id=…)` said to supply the missing half of the feature pair, and the corrected call then failed on the conflict with `comid`; `nwdc` responded to `state=[]` by saying exactly one of `state`, `county` or `huc` must be given, when exactly one was; `codes.states.apply_state` offered NGWMN callers a `state_name` / `state_code` parameter no NGWMN getter accepts; and `BaseMetadata` pointed NGWMN and NWDC callers at a Water Data getter. **Bug fix:** `nldi.get_features(navigation_mode=…)` without a `data_source` wrote `None` into the URL path and returned an empty frame from a 200; it now raises and names the source to pass. **Behavior change:** `waterdata.get_nearest_continuous` appends `time` and `monitoring_location_id` to an explicit `properties` list rather than honoring it verbatim -- omitting either silently collapsed every monitoring location into one row per target, a wrong answer rather than an error -- so a caller who passed `properties=['time', 'value']` now gets a third column. **Behavior change:** `nwis.query_waterdata` serves `'peaks'` only; the `'ratings'` URL it used to build was never an NwisWeb program and responded with an HTML error page. `nwis.get_record(service='ratings')` is unaffected -- it routes to `get_ratings`, which is served from a different endpoint. New: `waterdata.get_cql(..., max_rows=N)` caps the total rows a CQL query returns. +**08/20/2026:** Argument checks now share one vocabulary, and every rejection explains how to correct the call. `dataretrieval._validation` owns the message shape for each check that recurs across the adapters -- a value outside a closed vocabulary (`require_one_of`), a missing argument (`require_argument`, `require_together`), a query with no filter at all (`require_any_of`), and arguments that conflict (`require_exactly_one`, `reject_together`) -- so a new check cannot invent its own phrasing, which is how `get_reference_table` came to tell callers who passed a bad `collection` that their *code service* was invalid. Each check takes the caller's own name for the parameter and a remedy for the action it cannot derive, and every check raises `ValueError` -- one class for a bad argument value. **Behavior change:** the text of those rejections moves with them -- `"Unrecognized service: 'x'. get_record serves …"` is now `"Invalid service: 'x'. Valid options are: …"`, and the major-filter and bounding-box rejections in `query_waterdata` / `query_waterservices` are rendered in the shared form. **Behavior change:** the deprecated `nwis` query entry points (`query_waterdata`, `query_waterservices`, `get_record`) now respond to a missing major filter, an incomplete bounding box, or an unknown service with `ValueError` rather than their historic `TypeError` -- `TypeError` remains for a mistyped argument, such as a non-string `sites`. Code catching `TypeError` there, or matching on the old strings, must update. **Bug fix:** a `None` passed as a major filter (`query_waterservices(service='dv', sites=None)`) counted as a filter and reached the service as an empty `sites=`; `None` now means not supplied, and the call is refused with the filters that would have been accepted. **Bug fix:** four messages told callers to do something that raised again or named a parameter their getter does not accept -- `nldi.get_features(comid=…, feature_id=…)` said to supply the missing half of the feature pair, and the corrected call then failed on the conflict with `comid`; `nwdc` responded to `state=[]` by saying exactly one of `state`, `county` or `huc` must be given, when exactly one was; `codes.states.apply_state` offered NGWMN callers a `state_name` / `state_code` parameter no NGWMN getter accepts; and `BaseMetadata` directed NGWMN and NWDC callers to a Water Data getter. **Bug fix:** `nldi.get_features(navigation_mode=…)` without a `data_source` wrote `None` into the URL path and returned an empty frame from a 200; it now raises and names the source to pass. **Behavior change:** `waterdata.get_nearest_continuous` appends `time` and `monitoring_location_id` to an explicit `properties` list rather than using it verbatim -- omitting either collapsed every monitoring location into one row per target, a wrong result rather than an error -- so a caller who passed `properties=['time', 'value']` now gets a third column. **Behavior change:** `nwis.query_waterdata` serves `'peaks'` only; the `'ratings'` URL it used to build was never an NwisWeb program and responded with an HTML error page. `nwis.get_record(service='ratings')` is unaffected -- it routes to `get_ratings`, which is served from a different endpoint. New: `waterdata.get_cql(..., max_rows=N)` caps the total rows a CQL query returns. -**08/13/2026:** Warning categories now say what they mean. Two advisories about *upstream data* were emitted as `DeprecationWarning` and as an uncategorized `warnings.warn` respectively; both are now `DataCurrencyWarning` (a `UserWarning` subclass, exported as `dataretrieval.DataCurrencyWarning`). **Behavior change:** WQP's legacy-WQX notice moves from `DeprecationWarning` to `DataCurrencyWarning`. Because `legacy=True` is the default on every WQP getter and the notice is unconditional, a downstream project running `-W error::DeprecationWarning` previously could not call any WQP getter with default arguments; it now can. The cost is visibility — `DeprecationWarning` is silent by default outside `__main__`, so this notice will now print to stderr for library and notebook callers who never saw it. Suppress it with `warnings.filterwarnings("ignore", category=dataretrieval.DataCurrencyWarning)`, or set `legacy=False` where a WQX3.0 profile exists. The NWIS qw-endpoint retirement notice gains the same category (it previously had none, arriving as a bare `UserWarning` that deprecation filters ignored). `DeprecationWarning` now means only that a name in this package is being removed, always with a replacement and, where published, a removal date; those horizons are declared once in `dataretrieval._deprecation.REMOVALS` rather than declared at each call site. +**08/13/2026:** Warning categories now match their meaning. Two advisories about *upstream data* were emitted as `DeprecationWarning` and as an uncategorized `warnings.warn` respectively; both are now `DataCurrencyWarning` (a `UserWarning` subclass, exported as `dataretrieval.DataCurrencyWarning`). **Behavior change:** WQP's legacy-WQX notice moves from `DeprecationWarning` to `DataCurrencyWarning`. Because `legacy=True` is the default on every WQP getter and the notice is unconditional, a downstream project running `-W error::DeprecationWarning` previously could not call any WQP getter with default arguments; it now can. The cost is visibility — `DeprecationWarning` is silent by default outside `__main__`, so this notice will now print to stderr for library and notebook callers who never saw it. Suppress it with `warnings.filterwarnings("ignore", category=dataretrieval.DataCurrencyWarning)`, or set `legacy=False` where a WQX3.0 profile exists. The NWIS qw-endpoint retirement notice gains the same category (it previously had none, emitted as a bare `UserWarning` that deprecation filters ignored). `DeprecationWarning` now means only that a name in this package is being removed, always with a replacement and, where published, a removal date; those horizons are declared once in `dataretrieval._deprecation.REMOVALS` rather than declared at each call site. -**08/11/2026:** Settings resolve through a layered chain instead of the environment alone, and a *configuration profile* is a named set of settings for **one adapter**. The new `dataretrieval.configuration` module resolves every setting in one order, highest first: a configuration passed to an active `dataretrieval.configure(...)` block, a profile that block selected, the setting's `API_USGS_*` environment variable, the adapter's `[]` table in `~/.dataretrieval/config.toml` (or `DATARETRIEVAL_CONFIG`), the file's top-level keys, the adapter's own built-in preference, then the package default. Precedence applies **per setting**, so a file that sets only `concurrency` leaves an environment `API_USGS_PAT` in effect, and a `[ngwmn]` table still inherits every top-level key it does not name. `configure()` takes configuration objects positionally, at most one per adapter and nothing else: `configure(Configuration(api_key=vault.read("usgs/pat")), WaterdataConfiguration.load("bulk"), NgwmnConfiguration(concurrency=4))`. The adapter a configuration targets is a property of its class, so a caller never restates it, and each adapter owns its class in the module that *reads* those settings (`waterdata.WaterdataConfiguration`, `ngwmn.NgwmnConfiguration`, `nwdc.NwdcConfiguration`, `wqp.WqpConfiguration`, `nldi.NldiConfiguration`, `streamstats.StreamstatsConfiguration`) — an adapter accepts only the settings it reads, so `[streamstats] parallel_chunks = 8` is an error rather than a line that quietly does nothing. The block is delivered through a `ContextVar`, so a credential set inside it cannot leak across threads or asyncio tasks, which is what makes it safe for a server or notebook handling several users' keys and is the thing assigning to `os.environ` could never do (issue #352). The file gains named profiles beside each adapter's default profile: `[waterdata]` is always in effect, `[waterdata.bulk]` only when a caller selects it with `WaterdataConfiguration.load("bulk")`, and a selected profile still inherits the default profile and the package-wide keys per setting. A profile named in code outranks the setting's environment variable — the one place the ladder inverts the environment-above-file rule, because a stale shell export overriding a deliberate selection would look like a bug. An adapter's configuration may also include a `base_url`, which redirects that adapter's requests for the duration of the block — a staging instance, a mirror, a recording proxy — and no other adapter's; for Water Data one value moves the OGC collections, the Samples database, the statistics service and the STAC catalog together. It is settable in a `configure()` block only: a `base_url` key in the file and an exported `API_USGS_BASE_URL` each raise rather than being read, since a redirect a config file or a shell profile can set is one no reader of the script can see. The API key does not follow a redirect — it is scoped to the single host that honors it — and is deliberately not per-adapter: it authenticates to the gateway fronting a host, and Water Data and NGWMN share that host, one key, and one hourly quota. `dataretrieval.show_configuration()` reports each setting's effective value and where it came from, naming the profile behind each value (`configure() block [waterdata.bulk]` rather than a bare block), listing the profiles a file defines whether or not this run selected any, and naming any adapter this process has not imported rather than omitting it — without ever printing the key. One parser per setting owns its grammar, so a value means the same thing whichever source wrote it. **Breaking change:** `RetryPolicy.from_env()` is now `RetryPolicy.from_configuration()` and resolves through the whole chain rather than the environment alone. **Behavior change:** a credential-shaped keyword passed to a getter's `**kwargs` query passthrough — Water Data's `**queryables` and every WQP getter's search filters — now raises `TypeError` naming `configure(Configuration(api_key=...))` instead of putting a secret in a URL that clients, proxies and logs retain. The names refused are `api_key=`, `token=`, `x_api_key=`, `password=`, `auth=`, `pat=` and similar names; a filter the server actually defines is unaffected. **Bug fix:** `API_USGS_STALL_TIMEOUT` was read straight from `os.environ`, so it could not be set by a `configure()` block or the config file and never appeared in `show_configuration()`; it now resolves through the chain like every other setting. Rationale in ADRs 0009, 0010 and 0011; terms in `CONTEXT.md`. +**08/11/2026:** Settings resolve through a layered chain instead of the environment alone, and a *configuration profile* is a named set of settings for **one adapter**. The new `dataretrieval.configuration` module resolves every setting in one order, highest first: a configuration passed to an active `dataretrieval.configure(...)` block, a profile that block selected, the setting's `API_USGS_*` environment variable, the adapter's `[]` table in `~/.dataretrieval/config.toml` (or `DATARETRIEVAL_CONFIG`), the file's top-level keys, the adapter's own built-in preference, then the package default. Precedence applies **per setting**, so a file that sets only `concurrency` leaves an environment `API_USGS_PAT` in effect, and a `[ngwmn]` table still inherits every top-level key it does not name. `configure()` takes configuration objects positionally, at most one per adapter and nothing else: `configure(Configuration(api_key=vault.read("usgs/pat")), WaterdataConfiguration.load("bulk"), NgwmnConfiguration(concurrency=4))`. The adapter a configuration targets is a property of its class, so a caller never restates it, and each adapter owns its class in the module that *reads* those settings (`waterdata.WaterdataConfiguration`, `ngwmn.NgwmnConfiguration`, `nwdc.NwdcConfiguration`, `wqp.WqpConfiguration`, `nldi.NldiConfiguration`, `streamstats.StreamstatsConfiguration`) — an adapter accepts only the settings it reads, so `[streamstats] parallel_chunks = 8` is an error rather than a line that does nothing. The block is delivered through a `ContextVar`, so a credential set inside it cannot leak across threads or asyncio tasks, which is what makes it safe for a server or notebook handling several users' keys and is the thing assigning to `os.environ` could never do (issue #352). The file gains named profiles beside each adapter's default profile: `[waterdata]` is always in effect, `[waterdata.bulk]` only when a caller selects it with `WaterdataConfiguration.load("bulk")`, and a selected profile still inherits the default profile and the package-wide keys per setting. A profile named in code outranks the setting's environment variable — the one place the ladder inverts the environment-above-file rule, because a stale shell export overriding a deliberate selection would look like a bug. An adapter's configuration may also include a `base_url`, which redirects that adapter's requests for the duration of the block — a staging instance, a mirror, a recording proxy — and no other adapter's; for Water Data one value redirects the OGC collections, the Samples database, the statistics service and the STAC catalog together. It is settable in a `configure()` block only: a `base_url` key in the file and an exported `API_USGS_BASE_URL` each raise rather than being read, since a redirect a config file or a shell profile can set is one no reader of the script can see. The API key does not follow a redirect — it is scoped to the single host that accepts it — and is deliberately not per-adapter: it authenticates to the gateway fronting a host, and Water Data and NGWMN share that host, one key, and one hourly quota. `dataretrieval.show_configuration()` reports each setting's effective value and where it came from, naming the profile behind each value (`configure() block [waterdata.bulk]` rather than a bare block), listing the profiles a file defines whether or not this run selected any, and naming any adapter this process has not imported rather than omitting it — without ever printing the key. One parser per setting owns its grammar, so a value means the same thing whichever source wrote it. **Breaking change:** `RetryPolicy.from_env()` is now `RetryPolicy.from_configuration()` and resolves through the whole chain rather than the environment alone. **Behavior change:** a credential-shaped keyword passed to a getter's `**kwargs` query passthrough — Water Data's `**queryables` and every WQP getter's search filters — now raises `TypeError` naming `configure(Configuration(api_key=...))` instead of putting a secret in a URL that clients, proxies and logs retain. The names refused are `api_key=`, `token=`, `x_api_key=`, `password=`, `auth=`, `pat=` and similar names; a filter the server defines is unaffected. **Bug fix:** `API_USGS_STALL_TIMEOUT` was read straight from `os.environ`, so it could not be set by a `configure()` block or the config file and never appeared in `show_configuration()`; it now resolves through the chain like every other setting. Rationale in ADRs 0009, 0010 and 0011; terms in `CONTEXT.md`. -**08/11/2026:** `dataretrieval.wateruse` is now `dataretrieval.nwdc`. Every other adapter is named for the service it retrieves from — `ngwmn`, `nldi`, `wqp`, `streamstats`, `nwis` — and this one was named for one subset of what its service offers. The National Water Availability Assessment Data Companion serves ten modeled datasets; the water-use models are five of them, the rest being hydrologic, atmospheric-forcing, and assessment outputs (`GET https://api.water.usgs.gov/nwaa-data/models`). **Deprecation:** `dataretrieval.wateruse` still works and re-exports `dataretrieval.nwdc` unchanged, emitting a `DeprecationWarning` on import; it will be removed on or after 2027-08-11. The alias forwards rather than copies, so `wateruse.get_wateruse is nwdc.get_wateruse` — monkeypatching or identity comparison through either name behaves the same. `import dataretrieval` stays silent: the package imports `nwdc` directly, so only code naming `wateruse` itself sees the warning. Function and constant names are unchanged (`get_wateruse`, `MODELS`, `WATERUSE_URL`, `DEFAULT_CONCURRENT_REQUESTS`). Terms are defined in `CONTEXT.md`. +**08/11/2026:** `dataretrieval.wateruse` is now `dataretrieval.nwdc`. Every other adapter is named for the service it retrieves from — `ngwmn`, `nldi`, `wqp`, `streamstats`, `nwis` — and this one was named for one subset of what its service offers. The National Water Availability Assessment Data Companion serves ten modeled datasets; the water-use models are five of them, the rest being hydrologic, atmospheric-forcing, and assessment outputs (`GET https://api.water.usgs.gov/nwaa-data/models`). **Deprecation:** `dataretrieval.wateruse` still works and re-exports `dataretrieval.nwdc` unchanged, emitting a `DeprecationWarning` on import; it will be removed on or after 2027-08-11. The alias forwards rather than copies, so `wateruse.get_wateruse is nwdc.get_wateruse` — monkeypatching or identity comparison through either name behaves the same. `import dataretrieval` emits no warning: the package imports `nwdc` directly, so only code naming `wateruse` itself sees the warning. Function and constant names are unchanged (`get_wateruse`, `MODELS`, `WATERUSE_URL`, `DEFAULT_CONCURRENT_REQUESTS`). Terms are defined in `CONTEXT.md`. -**08/09/2026:** `waterdata.get_cql` takes `collection` rather than `service`. OGC API - Features (17-069r4) normatively names this value the `collectionId`: Requirement 20 fixes the path template `/collections/{collectionId}/items`, and Requirement 18 defines `collectionId` as each `id` in the collections response -- which is literally how the package builds the URL, and what the live API returns. *Service* names the API itself (Water Data, NGWMN). **Deprecation:** `service=` still works and resolves to `collection`, with a `DeprecationWarning`; it will be removed on or after 2027-08-09. Positional callers (`get_cql("daily", cql)`) are unaffected. The `WATERDATA_SERVICES` type alias is now `WATERDATA_COLLECTIONS`, with `WATERDATA_SERVICES` retained as a permanent alias for the same object. Terms are defined in `CONTEXT.md`. +**08/09/2026:** `waterdata.get_cql` takes `collection` rather than `service`. OGC API - Features (17-069r4) normatively names this value the `collectionId`: Requirement 20 fixes the path template `/collections/{collectionId}/items`, and Requirement 18 defines `collectionId` as each `id` in the collections response -- which is how the package builds the URL, and what the live API returns. *Service* names the API itself (Water Data, NGWMN). **Deprecation:** `service=` still works and resolves to `collection`, with a `DeprecationWarning`; it will be removed on or after 2027-08-09. Positional callers (`get_cql("daily", cql)`) are unaffected. The `WATERDATA_SERVICES` type alias is now `WATERDATA_COLLECTIONS`, with `WATERDATA_SERVICES` retained as a permanent alias for the same object. Terms are defined in `CONTEXT.md`. -**08/09/2026:** Every retrieval path now runs through one executor. `waterdata.get_cql` (via the OGC `fetch_ogc_request`) and `waterdata.get_stats_por` / `get_stats_date_range` (via the Statistics page walk) previously bypassed `dataretrieval.transport.fanout.FanOut` through a private sync bridge, which meant they were the only getters in the package with **no retry**: a mid-page-walk 429 or 503 failed the whole call while every typed getter and Water Use retried through it. Both now run as a one-item fan-out and the 25-line `transport/sync.py` is gone. **Behavior change:** those three getters now retry transient failures (`API_USGS_RETRIES`, default 4) and, when the retries are exhausted, raise the resumable `ServiceInterrupted` / `QuotaExhausted` rather than `ServiceUnavailable` / `RateLimited` / `NetworkError` — all remain `DataRetrievalError`, so broad handlers are unaffected, but narrow handlers around those calls must widen, and `.call.resume()` is now available on the interruption. A failure that retrying cannot fix (bad scheme, a hostname that does not resolve) is still raised as `NetworkError` immediately. The progress line moved with it: `FanOut.resume()` opens the reporter it ticks into, so a driver can no longer run the shared executor and silently print nothing, and a `.call.resume()` invoked long after the interruption now reports progress instead of printing nothing. Internal tidying with no public effect: the WQX3 / legacy-WQP CSV datetime shaping moved out of `dataretrieval.utils` (whose docstring reserves it for non-service-specific shaping) into the `dataretrieval._wqx` leaf; the five Water Data endpoint URLs are declared once in `dataretrieval.waterdata.endpoints` instead of being derived in three modules; the OGC queryables document is parsed by `dataretrieval.ogc.schema` so every OGC adapter can offer the table, with `waterdata.get_queryables` unchanged as its documented wrapper; and `ogc/engine.py` imports each symbol from the module that defines it. +**08/09/2026:** Every retrieval path now runs through one executor. `waterdata.get_cql` (via the OGC `fetch_ogc_request`) and `waterdata.get_stats_por` / `get_stats_date_range` (via the Statistics page walk) previously bypassed `dataretrieval.transport.fanout.FanOut` through a private sync bridge, which meant they were the only getters in the package with **no retry**: a mid-page-walk 429 or 503 failed the whole call while every typed getter and Water Use retried through it. Both now run as a one-item fan-out and the 25-line `transport/sync.py` is gone. **Behavior change:** those three getters now retry transient failures (`API_USGS_RETRIES`, default 4) and, when the retries are exhausted, raise the resumable `ServiceInterrupted` / `QuotaExhausted` rather than `ServiceUnavailable` / `RateLimited` / `NetworkError` — all remain `DataRetrievalError`, so broad handlers are unaffected, but narrow handlers around those calls must widen, and `.call.resume()` is now available on the interruption. A failure that retrying cannot fix (bad scheme, a hostname that does not resolve) is still raised as `NetworkError` immediately. The progress line moved with it: `FanOut.resume()` opens the reporter it ticks into, so a driver can no longer run the shared executor and print nothing, and a `.call.resume()` invoked long after the interruption now reports progress instead of printing nothing. Internal reorganization with no public effect: the WQX3 / legacy-WQP CSV datetime shaping moved out of `dataretrieval.utils` (whose docstring reserves it for non-service-specific shaping) into the `dataretrieval._wqx` leaf; the five Water Data endpoint URLs are declared once in `dataretrieval.waterdata.endpoints` instead of being derived in three modules; the OGC queryables document is parsed by `dataretrieval.ogc.schema` so every OGC adapter can return the table, with `waterdata.get_queryables` unchanged as its documented wrapper; and `ogc/engine.py` imports each symbol from the module that defines it. **08/09/2026:** Internal structure cleanup, no public API change. Validating a server-supplied next-page link is now one policy in `dataretrieval.transport.links` instead of three divergent copies (the OGC engine, the ratings STAC walk, and Water Use). Two of those copies were fixed by the merge: the OGC page walk now resolves a *relative* `next` href against the page it came from (it previously returned the unresolved reference as the pagination cursor) and refuses an unparseable one rather than following it unchecked. Cross-host refusal, credential stripping, and Water Use's host-alias rewrite are unchanged, as is the error type each walk raises. `parse_retry_after` moved to `dataretrieval.exceptions`, next to the `DataRetrievalError.retry_after` field it exists to produce. The one-shot HTTP query path (`query`, `to_str`, and their helpers) moved out of `dataretrieval.utils` into the private `dataretrieval._querying`; `dataretrieval.utils.query` and `dataretrieval.utils.to_str` remain the documented public paths, as `Ambient` and `BaseMetadata` already do. `waterdata` profile validation moved next to the tables it validates in `waterdata.types`, and `nwis.get_dv`/`get_iv` now share one body. **08/06/2026:** Fan-out execution is now shared across services. Chunking is how a query is divided structurally (a Water Data/NGWMN URL over the byte limit); fan-out is how the pieces are distributed operationally. Only the first is protocol-specific, so the executor moved to `dataretrieval.transport.fanout` (`FanOut`, over a three-member `FanOutPlan` protocol) while chunk planning stays in `dataretrieval.ogc`. Water Use no longer re-implements the fan-out gather and inherits resume, progress reporting, and `API_USGS_CONCURRENT`: a multi-location pull interrupted by a rate limit now raises a resumable interruption whose `.call.resume()` re-issues only the locations that did not finish, instead of discarding every completed one. The interruption taxonomy moved to the `dataretrieval.interruptions` leaf and its base class is now `FanOutInterrupted`; **`ChunkInterrupted` is a permanent alias of the same class**, so `except ChunkInterrupted` keeps working. **Breaking change:** a Water Use fan-out interrupted by a 5xx, 429, or recoverable connection failure now raises `ServiceInterrupted`/`QuotaExhausted` rather than `ServiceUnavailable`/`RateLimited`/`NetworkError` — all remain `DataRetrievalError`, so broad handlers are unaffected, but narrow handlers around a Water Use call must widen. **Breaking change:** `wateruse.MAX_CONCURRENT_REQUESTS` is removed; set `API_USGS_CONCURRENT` (which now outranks any service default) or read `wateruse.DEFAULT_CONCURRENT_REQUESTS`. -**08/03/2026:** Split the Water Data implementation into focused time-series, metadata, measurements, reference, samples, and CQL collection-family modules behind the unchanged `waterdata.api` facade. Active service modules now declare explicit exports; public Water Data imports, signatures, function identities, deprecations, and return contracts are protected by executable contract snapshots. OGC ambient context and schema/queryables execution are separated from request construction, adapter-to-adapter reach-through is prohibited by architecture tests, and service-specific output shapes are documented rather than forced into one model. +**08/03/2026:** Split the Water Data implementation into focused time-series, metadata, measurements, reference, samples, and CQL collection-family modules behind the unchanged `waterdata.api` facade. Active service modules now declare explicit exports; public Water Data imports, signatures, function identities, deprecations, and return contracts are protected by executable contract snapshots. OGC ambient context and schema/queryables execution are separated from request construction, adapter-to-adapter imports are prohibited by architecture tests, and service-specific output shapes are documented rather than forced into one model. **08/02/2026:** Added an internal API-neutral transport layer for guarded HTTP clients, host-scoped authentication, cursor pagination, bounded retry, response aggregation, progress, and sync-over-async dispatch. Water Use and the non-OGC Statistics API now consume transport directly instead of private OGC execution helpers; WQP, NLDI, and StreamStats opt into bounded transient retry while deprecated NWIS behavior remains unchanged. OGC retains CQL2, request construction, feature shaping, chunk planning, resumable calls, and interruption types, with compatibility imports at previous private paths. Failed pagination and fan-out still raise rather than returning partial data, and no public signatures or return shapes changed. -**08/02/2026:** Phase 1 OGC boundary stabilization: the `dataretrieval.ogc` package now exposes a deliberate, small facade (`OgcDialect`, `prepare_request_args`, `get_ogc_data`, `fetch_ogc_request`) that service adapters (NGWMN, Water Data's generic wrapper) import. Internal request-construction helpers moved to a new `ogc.requests` module; the dialect type and endpoint constants live in the leaf `ogc.policy` module. `ogc.shaping` no longer depends on `ogc.engine` at all, and the complete runtime OGC import graph is now acyclic. `_default_headers` now accepts a target URL and adds `X-Api-Key` only for `api.waterdata.usgs.gov`; shared sync and async clients also strip the key before following any cross-host redirect, including external rating-asset downloads. `waterdata.utils` no longer bulk-re-exports private OGC symbols, and consumers import implementation helpers from their canonical modules. No public API, return-value, or deprecation changes. +**08/02/2026:** Phase 1 OGC boundary stabilization: the `dataretrieval.ogc` package now exposes a deliberate, small facade (`OgcDialect`, `prepare_request_args`, `get_ogc_data`, `fetch_ogc_request`) that service adapters (NGWMN, Water Data's generic wrapper) import. Internal request-construction helpers moved to a new `ogc.requests` module; the dialect type and endpoint constants are defined in the leaf `ogc.policy` module. `ogc.shaping` no longer depends on `ogc.engine` at all, and the complete runtime OGC import graph is now acyclic. `_default_headers` now accepts a target URL and adds `X-Api-Key` only for `api.waterdata.usgs.gov`; shared sync and async clients also strip the key before following any cross-host redirect, including external rating-asset downloads. `waterdata.utils` no longer bulk-re-exports private OGC symbols, and consumers import implementation helpers from their canonical modules. No public API, return-value, or deprecation changes. **08/02/2026:** Fixed source-distribution and wheel package discovery so the `dataretrieval.ogc` and `dataretrieval.waterdata` subpackages are included in installed artifacts. CI now builds and installs the wheel outside the source checkout before importing the core service modules. Added an architecture baseline, initial decision records, and executable dependency-direction checks for the existing modular-monolith boundaries. @@ -38,23 +38,23 @@ **06/03/2026:** The request-error hierarchy is now unified. Every module (`nwis`, `wqp`, `nldi`, `waterdata`, `nadp`, `streamstats`) raises a subclass of `dataretrieval.DataRetrievalError` on a failed request, so a single `except dataretrieval.DataRetrievalError` spans them all. An HTTP error status is raised as an `HTTPError` with `.status_code` (inspect it to branch on a specific code); the retryable 429/5xx subset is `TransientError` (`RateLimited` / `ServiceUnavailable`, with `.retry_after`); and a request too large to satisfy is a `RequestTooLarge` (`URLTooLong` for an over-long single request, `Unchunkable` when the Water Data chunker cannot split a call small enough). Connection-level failures (timeouts, DNS, refused connections) are wrapped as a `NetworkError`, with the underlying `httpx` exception on `__cause__`. Every `DataRetrievalError` also exposes `.status_code` (`None` when there is no HTTP status), `.retry_after`, and `.retryable`, so a single `except dataretrieval.DataRetrievalError as e` clause can branch on the status or retry transient failures without knowing the concrete subclass. **Breaking change:** these exceptions no longer multiply-inherit a built-in — code that caught request failures with `except ValueError` or `except RuntimeError` should switch to `except dataretrieval.DataRetrievalError` (or a specific subclass). A no-data result is **not** an error: the modern getters (`waterdata`, `wqp`, `nldi`) return an empty DataFrame when nothing matches. Only the deprecated `nwis` (waterservices) path still raises `NoSitesError` on no data. -**05/17/2026:** The OGC `waterdata` getters (`get_daily`, `get_continuous`, `get_field_measurements`, and the rest of the multi-value-capable functions) now transparently chunk requests whose URLs would otherwise exceed the server's ~8 KB byte limit. +**05/17/2026:** The OGC `waterdata` getters (`get_daily`, `get_continuous`, `get_field_measurements`, and the rest of the multi-value-capable functions) now automatically chunk requests whose URLs would otherwise exceed the server's ~8 KB byte limit. -**05/16/2026:** Fixed silent truncation in the paginated `waterdata` request loops (`_walk_pages` and `get_stats_data`). Mid-pagination failures (HTTP 429, 5xx, network error) were previously caught and ignored — pagination would quietly stop and the function would return whatever rows it had collected, leaving callers with truncated DataFrames they had no way to detect. The loops now status-check every page like the initial request and raise `RuntimeError` on any failure, with the upstream exception chained as `__cause__` and a short list of recovery actions (wait and retry, reduce the request, or obtain an API token) in the message. **Behavior change**: callers that previously consumed partial DataFrames on transient upstream failures will now see an exception; retry the call (possibly with a smaller `limit` or narrower query). +**05/16/2026:** Fixed undetected truncation in the paginated `waterdata` request loops (`_walk_pages` and `get_stats_data`). Mid-pagination failures (HTTP 429, 5xx, network error) were previously caught and ignored — pagination would stop and the function would return whatever rows it had collected, leaving callers with truncated DataFrames they had no way to detect. The loops now status-check every page like the initial request and raise `RuntimeError` on any failure, with the upstream exception chained as `__cause__` and a short list of recovery actions (wait and retry, reduce the request, or obtain an API token) in the message. **Behavior change**: callers that previously consumed partial DataFrames on transient upstream failures will now see an exception; retry the call (possibly with a smaller `limit` or narrower query). -**05/07/2026:** Bumped the declared minimum Python version from **3.8** to **3.9** (`pyproject.toml`'s `requires-python` and the ruff target). This brings the manifest in line with what was already being tested — CI's matrix has long covered only 3.9, 3.13, and 3.14, the `waterdata` test module already skipped itself on Python < 3.10, and several modules already use 3.9-only stdlib (e.g. `zoneinfo`). Users on 3.8 will no longer be able to install the package; please upgrade. +**05/07/2026:** Bumped the declared minimum Python version from **3.8** to **3.9** (`pyproject.toml`'s `requires-python` and the ruff target). This makes the manifest match what was already tested — CI's matrix has long covered only 3.9, 3.13, and 3.14, the `waterdata` test module already skipped itself on Python < 3.10, and several modules already use 3.9-only stdlib (e.g. `zoneinfo`). Users on 3.8 will no longer be able to install the package; please upgrade. -**05/07/2026:** `waterdata.get_samples()` and `wqp.get_results()` now append a derived `DateTime` UTC column for every Date/Time/TimeZone triplet in the response (e.g. `Activity_StartDate` + `Activity_StartTime` + `Activity_StartTimeZone` → `Activity_StartDateTime`). Both the WQX3 (`Date`/`Time`/`TimeZone`) and legacy WQP (`Date`/`Time/Time`/`Time/TimeZoneCode`) shapes are recognized; abbreviations like EST/EDT/CST/PST resolve to a UTC `Timestamp`, unknown codes resolve to `NaT`, and the original triplet columns are preserved. Returned rows are also now sorted by `Activity_StartDateTime` (or the legacy `ActivityStartDateTime`) — the underlying APIs return rows in an unstable order. Mirrors R's `create_dateTime` and end-of-pipeline sort. Closes #266. +**05/07/2026:** `waterdata.get_samples()` and `wqp.get_results()` now append a derived `DateTime` UTC column for every Date/Time/TimeZone triplet in the response (e.g. `Activity_StartDate` + `Activity_StartTime` + `Activity_StartTimeZone` → `Activity_StartDateTime`). Both the WQX3 (`Date`/`Time`/`TimeZone`) and legacy WQP (`Date`/`Time/Time`/`Time/TimeZoneCode`) shapes are recognized; abbreviations like EST/EDT/CST/PST resolve to a UTC `Timestamp`, unknown codes resolve to `NaT`, and the original triplet columns are preserved. Returned rows are also now sorted by `Activity_StartDateTime` (or the legacy `ActivityStartDateTime`) — the underlying APIs return rows in an unstable order. Matches R's `create_dateTime` and end-of-pipeline sort. Closes #266. **05/06/2026:** Each remaining active function in `dataretrieval.nwis` now emits a per-function `DeprecationWarning` naming the `waterdata` replacement to migrate to (visible the first time users call each getter). The `nwis` module is scheduled for removal on or after **2027-05-06**. -**05/06/2026:** Added `waterdata.get_ratings(...)` — wraps the new Water Data STAC catalog (`api.waterdata.usgs.gov/stac/v0/search`) for USGS stage-discharge rating curves. Returns parsed `exsa` / `base` / `corr` rating tables as a dict of DataFrames keyed by feature ID, or just the list of available STAC features when `download_and_parse=False`. Mirrors R's `read_waterdata_ratings`. +**05/06/2026:** Added `waterdata.get_ratings(...)` — wraps the new Water Data STAC catalog (`api.waterdata.usgs.gov/stac/v0/search`) for USGS stage-discharge rating curves. Returns parsed `exsa` / `base` / `corr` rating tables as a dict of DataFrames keyed by feature ID, or just the list of available STAC features when `download_and_parse=False`. Matches R's `read_waterdata_ratings`. -**05/06/2026:** Added `waterdata.get_field_measurements_metadata(...)` — wraps the OGC `field-measurements-metadata` collection. Returns one row per (location, parameter) field-measurement series describing its period of record, units, etc., without the underlying observations. Discrete-measurement analogue to `get_time_series_metadata`. Mirrors R's `read_waterdata_field_meta`. +**05/06/2026:** Added `waterdata.get_field_measurements_metadata(...)` — wraps the OGC `field-measurements-metadata` collection. Returns one row per (location, parameter) field-measurement series describing its period of record, units, etc., without the underlying observations. Discrete-measurement analogue to `get_time_series_metadata`. Matches R's `read_waterdata_field_meta`. **05/06/2026:** Added `waterdata.get_peaks(...)` — wraps the new OGC `peaks` collection, returning the annual peak streamflow / stage record for a monitoring location (one row per water year, per parameter). Standard input to flood-frequency analysis. Supports calendar/water-year filters and the usual location/parameter/CQL options shared with the other OGC getters. -**05/05/2026:** Added `waterdata.get_combined_metadata(...)` — wraps the Water Data API's `combined-metadata` collection, which joins the monitoring-locations catalog with the time-series-metadata catalog and returns one row per (location, parameter, statistic) inventory entry. This is the most flexible "what data is available" endpoint in the API: any location attribute (state, HUC, site type, drainage area, well-construction depth, …) can be combined with any time-series attribute (parameter code, statistic, data type, period of record, …) in a single query. Mirrors R's `read_waterdata_combined_meta`. +**05/05/2026:** Added `waterdata.get_combined_metadata(...)` — wraps the Water Data API's `combined-metadata` collection, which joins the monitoring-locations catalog with the time-series-metadata catalog and returns one row per (location, parameter, statistic) inventory entry. This is the most flexible inventory endpoint in the API: any location attribute (state, HUC, site type, drainage area, well-construction depth, …) can be combined with any time-series attribute (parameter code, statistic, data type, period of record, …) in a single query. Matches R's `read_waterdata_combined_meta`. **05/05/2026:** Added `waterdata.get_samples_summary(monitoringLocationIdentifier=...)` — wraps the Samples database `/summary/{id}` endpoint, returning per-characteristic result and activity counts plus first / most recent activity dates for a single monitoring location. Useful for taking inventory of available discrete-sample data before pulling observations with `get_samples`. @@ -68,9 +68,9 @@ - Added `get_reference_table` (and made it considerably simpler and faster in #209), then extended it to accept arbitrary collections-API query parameters (#214). - Removed the deprecated `waterwatch` module (#228) and several defunct NWIS stubs (#222, #225), and added `py.typed` so `dataretrieval` ships type information to downstream users (#186). - Now supports `pandas` 3.x (#221). -- The OGC `waterdata` getters (`get_continuous`, `get_daily`, `get_field_measurements`, and the six others built on the same OGC collections) now accept `filter` and `filter_lang` kwargs that are passed through to the service's CQL filter parameter. This enables advanced server-side filtering that isn't expressible via the other kwargs — most commonly, OR'ing multiple time ranges into a single request. A long expression made up of a top-level `OR` chain is transparently split into multiple requests that each fit under the server's URI length limit, and the results are concatenated. +- The OGC `waterdata` getters (`get_continuous`, `get_daily`, `get_field_measurements`, and the six others built on the same OGC collections) now accept `filter` and `filter_lang` kwargs that are passed through to the service's CQL filter parameter. This enables advanced server-side filtering that isn't expressible via the other kwargs — most commonly, OR'ing multiple time ranges into a single request. A long expression made up of a top-level `OR` chain is automatically split into multiple requests that each fit under the server's URI length limit, and the results are concatenated. -**12/04/2025:** The `get_continuous()` function was added to the `waterdata` module, which provides access to measurements collected via automated sensors at a high frequency (often 15 minute intervals) at a monitoring location. This is an early version of the continuous endpoint and should be used with caution as the API team improves its performance. In the future, we anticipate the addition of an endpoint(s) specifically for handling large data requests, so power users may want to delay heavy development using the new continuous endpoint. +**12/04/2025:** The `get_continuous()` function was added to the `waterdata` module, which provides access to measurements collected via automated sensors at a high frequency (often 15 minute intervals) at a monitoring location. This is an early version of the continuous endpoint and should be used with caution as the API team improves its performance. In the future, we anticipate the addition of an endpoint(s) specifically for handling large data requests, so power users may want to delay extensive development using the new continuous endpoint. **11/24/2025:** `dataretrieval` has a new module, `waterdata`, which gives users access to USGS's modernized [Water Data APIs](https://api.waterdata.usgs.gov/). The Water Data API endpoints include daily values, instantaneous values, field measurements (modernized groundwater levels service), time series metadata, and discrete water quality data from the Samples database. Though there will be a period of overlap, the functions within `waterdata` will eventually replace the `nwis` module, which currently provides access to the legacy [NWIS Water Services](https://waterservices.usgs.gov/). More example workflows and functions coming soon. Check `help(waterdata)` for more information. diff --git a/dataretrieval/__init__.py b/dataretrieval/__init__.py index 41b529cad..7e3349f75 100644 --- a/dataretrieval/__init__.py +++ b/dataretrieval/__init__.py @@ -27,9 +27,9 @@ reports what is in effect and where each value came from. A failed request raises a subclass of :class:`dataretrieval.DataRetrievalError` -(the taxonomy lives in ``dataretrieval.exceptions``); connection-level failures +(the taxonomy is defined in ``dataretrieval.exceptions``); connection-level failures (timeouts, DNS) are wrapped as :class:`dataretrieval.NetworkError`. A fanned-out -request interrupted mid-stream raises :class:`dataretrieval.FanOutInterrupted` +request interrupted partway raises :class:`dataretrieval.FanOutInterrupted` (also available under its original ``ChunkInterrupted`` name), whose ``.call.resume()`` continues from the work already completed. """ @@ -42,7 +42,7 @@ __version__ = "version-unknown" # Layered configuration: a ``with configure(...)`` block, the environment, then -# the config file. The canonical home is ``dataretrieval.configuration``; +# the config file. It is defined in ``dataretrieval.configuration``; # the callable is named ``configure`` so it doesn't shadow that module. # # The module itself is deliberately absent from ``__all__`` below: it and the @@ -70,8 +70,8 @@ # Resumable fan-out interruption exceptions. They are defined in # ``dataretrieval.interruptions`` rather than ``dataretrieval.exceptions`` # because they carry pandas/httpx state and a resumable ``FanOut`` handle, -# which would pull heavy dependencies into the lightweight exceptions module. -# They are not under ``ogc`` because Water Use raises them too. Surfaced here so +# which would make the exceptions module depend on pandas and httpx. +# They are not under ``ogc`` because Water Use raises them too. Re-exported here so # callers get a stable public path: ``from dataretrieval import ChunkInterrupted``. from dataretrieval.interruptions import ( ChunkInterrupted, @@ -81,7 +81,7 @@ ) # Parallel-chunks control (a context manager). Defined with the chunker in -# ``dataretrieval.ogc.chunking``; surfaced here for a stable public path +# ``dataretrieval.ogc.chunking``; re-exported here for a stable public path # ``from dataretrieval import parallel_chunks``. from dataretrieval.ogc.chunking import parallel_chunks @@ -97,7 +97,7 @@ ) __all__ = [ - # layered configuration (canonical home: ``dataretrieval.configuration``) + # layered configuration (defined in ``dataretrieval.configuration``) "Configuration", "configure", "show_configuration", @@ -110,7 +110,7 @@ "utils", "waterdata", "wqp", - # error taxonomy (canonical home: ``dataretrieval.exceptions``), re-exported + # error taxonomy (defined in ``dataretrieval.exceptions``), re-exported # so callers can ``except dataretrieval.DataRetrievalError`` "exceptions", "DataCurrencyWarning", diff --git a/dataretrieval/_ambient.py b/dataretrieval/_ambient.py index 05f9b0f60..745ecf744 100644 --- a/dataretrieval/_ambient.py +++ b/dataretrieval/_ambient.py @@ -13,7 +13,7 @@ class Ambient(Generic[_T]): """A :class:`~contextvars.ContextVar` paired with a scoping contextmanager. - Bundles the var and its set/reset-token dance into one object, so an ambient + Bundles the var and its set/reset-token handling into one object, so an ambient value needs a single declaration instead of a ``var`` + setter-function pair. Read the current value with :meth:`get`; set it for a ``with`` block by calling the instance. The previous value is restored on exit:: diff --git a/dataretrieval/_configuration_core.py b/dataretrieval/_configuration_core.py index 3e623ed35..62b3a6c42 100644 --- a/dataretrieval/_configuration_core.py +++ b/dataretrieval/_configuration_core.py @@ -1,6 +1,6 @@ """Private model, grammar, and file foundation for configuration. -The public interface and runtime precedence engine live in +The public interface and runtime precedence resolution are in ``dataretrieval.configuration``. This lower module keeps the mutually dependent configuration classes, setting grammar, TOML interpretation, and file caches together so the public facade can stay small without @@ -25,7 +25,7 @@ from dataretrieval._ambient import Ambient from dataretrieval.exceptions import ConfigurationError -#: Settings only an adapter can carry, because they name one service. No +#: Settings only an adapter can hold, because they name one service. No #: package-wide value could mean anything for them: there is no one base URL. #: #: The package-wide roster is :data:`SETTINGS`, declared below the class it is @@ -34,14 +34,14 @@ #: Environment variable backing a setting (precedence step 2). #: -#: Not every setting has one. ``parallel_chunks`` is deliberately absent: it -#: fans a query into more sub-requests, each of which spends rate-limit quota, -#: and ``dataretrieval.parallel_chunks`` documents why that must stay a -#: deliberate choice rather than a process-wide default. An environment -#: variable is process-wide and implicit -- exported once in a shell profile, -#: inherited by every subprocess, invisible at the call site. A config-file -#: entry is written deliberately and shows up in :func:`show_configuration`, so the -#: file and :func:`configure` block are the only sources for it. +#: Not every setting has one. ``parallel_chunks`` is deliberately absent: it splits a +#: query into more sub-requests, each of which spends rate-limit quota, and +#: ``dataretrieval.parallel_chunks`` documents why that must stay a deliberate choice +#: rather than a process-wide default. An environment variable is process-wide and +#: implicit -- exported once in a shell profile, inherited by every subprocess, +#: invisible at the call site. A config-file entry is written deliberately and appears +#: in :func:`show_configuration`, so the file and :func:`configure` block are the only +#: sources for it. ENV_VARS: dict[str, str] = { "api_key": "API_USGS_PAT", "concurrency": "API_USGS_CONCURRENT", @@ -50,12 +50,11 @@ "stall_timeout": "API_USGS_STALL_TIMEOUT", } -#: Variables the environment is *refused* for, by setting. Named rather than -#: left out of :data:`ENV_VARS`, so a caller who exports ``API_USGS_BASE_URL`` -#: gets an error instead of a silently ignored variable. The file refuses the -#: same key in the same words (:func:`_accepted_keys`): a base URL arriving -#: from outside the code could redirect the library to another host without a -#: reader of the script seeing it (ADR 0011). +#: Variables the environment is *refused* for, by setting. Named rather than left out of +#: :data:`ENV_VARS`, so a caller who exports ``API_USGS_BASE_URL`` gets an error instead +#: of an ignored variable. The file refuses the same key in the same words +#: (:func:`_accepted_keys`): a base URL set outside the code could redirect the library +#: to another host without a reader of the script seeing it (ADR 0011). #: #: Derived from :data:`ADAPTER_ONLY_SETTINGS` so the file and the environment #: cannot drift apart on which settings are code-only. @@ -71,7 +70,7 @@ #: The table ADR 0011 retired. Named here only so a file written against the #: earlier design gets an error that says what to write instead, rather than the -#: generic "unknown table" that would send the reader looking for a typo. +#: generic "unknown table" that would leave the reader looking for a typo. _RETIRED_PROFILES_TABLE = "profiles" #: Label for the file's top-level table, where keys are the defaults. @@ -80,12 +79,12 @@ #: Settings that warn when written at the top level of the file, and what to #: say. Declared as data, beside the other per-setting policies -- ``ENV_VARS``, #: ``_REFUSED_ENV_VARS``, ``_BLANK_MEANS_SET``, ``_VALIDATORS``, ``_DISPLAYS`` -- -#: so "what is special about ``parallel_chunks``?" is answerable from this block -#: rather than from a condition buried in a validation loop, and so a second +#: so what is special about ``parallel_chunks`` can be read from this block +#: rather than from a condition inside a validation loop, and so a second #: quota-spending setting is a row here rather than an edit to shared code. #: #: Top level only: a value in a ``[.]`` table is opt-in per run, -#: which is the shape a setting that spends quota wants. +#: which is the scope a setting that spends quota should have. _WARN_AT_TOP_LEVEL: dict[str, str] = { "parallel_chunks": ( f"'parallel_chunks' at {_TOP_LEVEL} applies to every query in every " @@ -106,7 +105,7 @@ # Values that turn the progress line off. Blank counts: ``API_USGS_PROGRESS=`` -# has always meant "off", not "unset" -- unlike the numeric knobs, where blank +# has always meant "off", not "unset" -- unlike the numeric settings, where blank # falls through to the default. _PROGRESS_FALSEY = frozenset({"", "0", "false", "no", "off"}) @@ -115,16 +114,16 @@ # setting a blank variable is what container and CI tooling produces when it # has nothing to pass (``docker run -e API_USGS_PAT``, a workflow secret that # is absent on a fork), so treating it as configured would let it shadow the -# config file and silently drop the user's API key. Keeping this a property of -# the setting -- rather than a second, lower visit to the environment -- keeps +# config file and drop the user's API key without an error. Keeping this a property of +# the setting -- rather than a second, lower-precedence read of the environment -- keeps # the chain in the shape the docstring and ADR 0009 describe. _BLANK_MEANS_SET = frozenset({"progress"}) -# Warnings about the config file report the file, not a call site: settings are -# resolved lazily from wherever a getter first needs one, so the user frame is -# a different depth every time and no fixed ``stacklevel`` can name it. Pointing -# at this module consistently at least makes the warnings filterable by module, -# and every message names the offending path and setting. +# Warnings about the config file report the file, not a call site: settings are resolved +# lazily from wherever a getter first needs one, so the user frame is a different depth +# every time and no fixed ``stacklevel`` can name it. Attributing them to this module +# consistently at least makes the warnings filterable by module, and every message names +# the offending path and setting. _WARN_STACKLEVEL = 2 _PROGRESS_TRUTHY = frozenset({"1", "true", "yes", "on"}) @@ -145,20 +144,19 @@ def __repr__(self) -> str: _UNSET: Any = _Unset() _SettingValue = str | None -# Overrides from the innermost active ``configure`` block, as raw strings so that -# every source shares one parser and one set of error messages. -# A package-wide override is keyed by the setting's name; an adapter-scoped one -# by ``(adapter, name)``. One flat mapping rather than a nested one so that -# nesting, per-key inheritance, and restore-on-exit keep falling out of a -# single merge, whichever scope a block sets. +# Overrides from the innermost active ``configure`` block, as raw strings so that every +# source shares one parser and one set of error messages. A package-wide override is +# keyed by the setting's name; an adapter-scoped one by ``(adapter, name)``. One flat +# mapping rather than a nested one so that nesting, per-key inheritance, and +# restore-on-exit all follow from a single merge, whichever scope a block sets. _ScopeKey = str | tuple[str, str] -# One frame per ``configure`` block, stacked outermost-first. Frames rather -# than a merged mapping: merged, an outer adapter-scoped block would beat an -# inner package-wide one, inverting the nesting rule ADR 0011 states. +# One frame per ``configure`` block, stacked outermost-first. Frames rather than a +# merged mapping: merged, an outer adapter-scoped block would outrank an inner +# package-wide one, inverting the nesting rule ADR 0011 states. # # Each entry pairs the raw value with the label naming where it came from, the # same shape the file source returns (:func:`_adapter_file_settings`). The -# label is built while the configuration object is still in hand, the only +# label is built while the configuration object is still available, the only # point where the *profile* is known -- by the time a value reaches the frame, # one from ``WaterdataConfiguration.load("bulk")`` and one from # ``WaterdataConfiguration(...)`` are indistinguishable. @@ -179,10 +177,10 @@ def __repr__(self) -> str: # Validated ``[]`` tables, keyed by adapter name and memoized on the # parsed file's identity, because an adapter table is validated only once that -# adapter is actually used. +# adapter is used. _adapter_cache: dict[str, tuple[_ParsedFile, Path, Mapping[str, tuple[str, str]]]] = {} -# Paths already warned about for loose permissions, so the warning fires once. +# Paths already warned about for loose permissions, so the warning is emitted once. _permission_warned: set[Path] = set() @@ -215,8 +213,8 @@ class _ParsedFile: # A setting means the same thing wherever it applies, but it does not apply # everywhere (ADR 0010). Each adapter declares the settings it accepts as the # fields of a ``BaseConfiguration`` subclass, defined *in the adapter's own -# module* (ADR 0011). Which settings an adapter accepts is the adapter's own -# knowledge; the setting itself is drawn from the shared groups below, so +# module* (ADR 0011). Which settings an adapter accepts is declared by the +# adapter; the setting itself is drawn from the shared groups below, so # ``retries`` is declared once. # # Two settings are deliberately absent from every adapter (ADR 0010): @@ -226,14 +224,14 @@ class _ParsedFile: #: Bound to the concrete subclass so ``WaterdataConfiguration.load(...)`` is #: typed as a ``WaterdataConfiguration`` rather than the base. ``typing.Self`` -#: would say this in one word and arrives in 3.11; the floor is 3.10. +#: would say this in one word and arrives in 3.11; the minimum is 3.10. _C = TypeVar("_C", bound="BaseConfiguration") #: Memoized :meth:`BaseConfiguration.settings` results, keyed on the class. #: A hand-rolled dict rather than ``functools.cache`` only because typeshed's #: wrapper takes ``Hashable`` and mypy does not accept a class for that -#: protocol, and this package carries no ``type: ignore``. +#: protocol, and this package contains no ``type: ignore``. _settings_cache: dict[type[BaseConfiguration], frozenset[str]] = {} @@ -241,12 +239,12 @@ def _settings_of(cls: type[BaseConfiguration]) -> frozenset[str]: """The setting names a configuration class accepts, computed once. A class constant in everything but spelling: the fields cannot change after - the class is created, and every adapter-scoped read asks for it -- through - :func:`_accepts`, before the frame walk and before the file. + the class is created, and every adapter-scoped read requests it -- through + :func:`_accepts`, before the scope stack is checked and before the file. Keyed on the *class* rather than on the adapter name because tests replace a registry entry to stand in for an unimported adapter; a name-keyed memo - would serve them the schema of the class they replaced. + would return the schema of the class they replaced. """ cached = _settings_cache.get(cls) if cached is None: @@ -263,8 +261,8 @@ class BaseConfiguration: an empty configuration is legal and one can be built up conditionally. Frozen, because a configuration is a value: two with the same settings are - interchangeable, and one already handed to :func:`configure` must not - change under the block that entered it. + interchangeable, and one already passed to :func:`configure` must not change while + the block that entered it is active. Values are checked when the configuration is *constructed*, so a typo raises where it was written rather than at a later ``with`` statement or @@ -275,7 +273,7 @@ class BaseConfiguration: #: caller imports. ``None`` on the package-wide :class:`Configuration`, #: which every adapter reads. A ``ClassVar``, not a field: the adapter is a #: property of the class, which is what stops the caller restating it at - #: every call site and stops the roster being spelled twice. + #: every call site and stops the roster being listed twice. adapter: ClassVar[str | None] = None #: The named profile these settings were read from, or ``None`` for a @@ -284,10 +282,10 @@ class BaseConfiguration: #: :func:`show_configuration` name the profile that supplied each value #: instead of reporting every block alike. #: - #: A ``ClassVar`` shadowed per instance by :meth:`load`, so it is neither a - #: field nor part of equality -- two configurations carrying the same - #: settings stay interchangeable however each was spelled, which is what - #: "a configuration is a value" means. + #: A ``ClassVar`` shadowed per instance by :meth:`load`, so it is neither a field + #: nor part of equality -- two configurations with the same settings stay + #: interchangeable however each was written, which is what "a configuration is a + #: value" means. profile: ClassVar[str | None] = None def __post_init__(self) -> None: @@ -301,7 +299,7 @@ def __post_init__(self) -> None: def validate(self) -> None: """Check rules that span more than one setting. - Does nothing by default. Per-setting grammar lives in this module's + Does nothing by default. Per-setting grammar is in this module's parsers and is shared with the file and the environment, so a value means the same thing whichever source wrote it; override this only for a rule no single setting can express. @@ -313,11 +311,11 @@ def settings(cls) -> frozenset[str]: return _settings_of(cls) def values(self) -> dict[str, Any]: - """The settings actually supplied, omitting those left unset. + """The settings supplied, omitting those left unset. An omitted setting inherits from an outer block or a lower source; an - explicit ``None`` suppresses them. Distinguishing the two is the whole - job of the ``_UNSET`` default, so it is done here rather than by every + explicit ``None`` suppresses them. Distinguishing the two is what the + ``_UNSET`` default exists for, so it is done here rather than by every reader. """ return { @@ -330,12 +328,12 @@ def values(self) -> dict[str, Any]: def load(cls: type[_C], profile: str) -> _C: """Read a named profile for this adapter from the configuration file. - ``[.]``. Only the keys that table names are carried, + ``[.]``. Only the keys that table names are included, so the profile still inherits the adapter's default profile and the package-wide keys per setting from the rungs below. Selecting a profile the file does not define raises: a name a caller - just typed is a typo worth reporting, not a silent fall-through to + just typed is a typo to report, not a fall-through to settings they did not ask for. Parameters @@ -346,7 +344,7 @@ def load(cls: type[_C], profile: str) -> _C: Returns ------- BaseConfiguration - An instance of the class it was called on, remembering the profile + An instance of the class it was called on, recording the profile it was read from so :func:`show_configuration` can name it. """ adapter = cls.adapter @@ -372,8 +370,8 @@ def _provenance(self) -> str: """How :func:`show_configuration` reports a value this supplied. The profile is named in the file's own spelling -- ``[waterdata.bulk]`` - -- so the report answers "which profile set this?" rather than only - "a block did", and the answer is greppable in the file that holds it. + -- so the report names the profile that set a value rather than only + the block, and the name can be searched for in the file that defines it. A configuration written in code has no profile, so it names its adapter alone; the package-wide one narrows to nothing and names neither. """ @@ -402,7 +400,7 @@ def _provenance(self) -> str: @dataclass(frozen=True) class _Retrying: - """Every adapter's retry dials: transient retries and the stall bound.""" + """Every adapter's retry settings: transient retries and the stall bound.""" retries: int | None = _UNSET stall_timeout: float | int | None = _UNSET @@ -410,7 +408,7 @@ class _Retrying: @dataclass(frozen=True) class _Redirectable: - """An adapter whose requests can be pointed at another base URL.""" + """An adapter whose requests can be sent to another base URL.""" base_url: str | None = _UNSET @@ -424,7 +422,9 @@ class _Concurrent: @dataclass(frozen=True) class _Chunked: - """An adapter whose queries divide into sub-requests the caller can fan.""" + """An adapter whose queries divide into sub-requests, fanned out further + at the caller's request. + """ parallel_chunks: int | None = _UNSET @@ -440,7 +440,7 @@ class Configuration(BaseConfiguration): Parameters ---------- api_key : str, optional - Water Data API key, sent as ``X-Api-Key`` and only ever to + Water Data API key, sent as ``X-Api-Key`` and only to ``api.waterdata.usgs.gov``. Prefer reading it from a secret store, the environment, or the configuration file over writing a literal into a script. Pass ``None`` to make a call without an ambient key. @@ -460,9 +460,9 @@ class Configuration(BaseConfiguration): for pulls you know are large. stall_timeout : float, optional Seconds a call may go without receiving *any* data before retrying - stops and the failure surfaces. Bounds the wall-clock cost of a dead + stops and the failure is raised. Bounds the wall-clock cost of a dead connection, which ``retries`` does not -- it counts attempts, not - seconds. Progress resets the clock; ``0`` disables the bound. + seconds. Progress resets the timer; ``0`` disables the bound. Examples -------- @@ -472,12 +472,11 @@ class Configuration(BaseConfiguration): df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000") """ - # Spelled out rather than composed from the groups above, because this - # order is also the order :func:`show_configuration` reports the settings - # in -- :data:`SETTINGS` is derived from it just below -- and composing - # would hand that reader-facing sequence to MRO linearization. The two - # adapter-only fields the groups carry are absent by construction here: - # there is no package-wide base URL. + # Listed rather than composed from the groups above, because this order is also the + # order :func:`show_configuration` reports the settings in -- :data:`SETTINGS` is + # derived from it just below -- and composing would leave that reader-facing + # sequence to MRO linearization. The two adapter-only fields the groups declare are + # absent by construction here: there is no package-wide base URL. api_key: str | None = _UNSET concurrency: int | str | None = _UNSET retries: int | None = _UNSET @@ -490,21 +489,20 @@ class Configuration(BaseConfiguration): #: them -- the fields of :class:`Configuration`, derived rather than restated. #: An adapter may accept a subset of them plus :data:`ADAPTER_ONLY_SETTINGS`. #: -#: Derived because the two copies had nothing holding them together, in the one -#: module whose job is to stop rosters being duplicated: a field added to the -#: class and forgotten here would work from :func:`configure` and be silently -#: dropped from the file -- :func:`_accepted_keys` would call it an unknown -#: setting -- and never appear in the report. Which is the "a schema no call -#: site can reach" failure ADR 0011 makes impossible by construction. This is -#: what the adapter side already does (:func:`settings_for`); only the -#: package-wide side was hand-maintained. +#: Derived because nothing checked that the two copies agreed, in the one module that +#: exists to stop rosters being duplicated: a field added to the class and forgotten +#: here would work from :func:`configure` and be dropped from the file without an error +#: -- :func:`_accepted_keys` would call it an unknown setting -- and never appear in the +#: report. That is the failure ADR 0011 makes impossible: a schema no call site can use +#: by construction. This is what the adapter side already does (:func:`settings_for`); +#: only the package-wide side was hand-maintained. #: #: Declared here, below the class, because it cannot be derived before the #: class exists. Every reader is a call-time lookup or a ``def`` default #: evaluated further down the module. SETTINGS: tuple[str, ...] = tuple(f.name for f in fields(Configuration)) -#: Every setting name this module knows a grammar for. +#: Every setting name this module has a grammar for. _ALL_SETTINGS: tuple[str, ...] = SETTINGS + ADAPTER_ONLY_SETTINGS @@ -512,11 +510,11 @@ class Configuration(BaseConfiguration): #: imports. Names only, because this module is a standard-library-only leaf #: every adapter may import and so cannot import them back. #: -#: Holding the names here rather than deriving them from the registry below is -#: what lets a ``[nldi]`` table stay valid in a file: NLDI is imported on demand -#: for the geopandas extra, so a roster built from imports would reject a valid -#: table until something happened to import that module, and the verdict would -#: vary by what a caller had touched. +#: Holding the names here rather than deriving them from the registry below is what lets +#: a ``[nldi]`` table stay valid in a file: NLDI is imported on demand for the geopandas +#: extra, so a roster built from imports would reject a valid table until something +#: happened to import that module, and the result would vary by what a caller had +#: imported. ADAPTERS: tuple[str, ...] = ( "waterdata", "ngwmn", @@ -527,7 +525,7 @@ class Configuration(BaseConfiguration): ) #: Configuration classes that have registered themselves, keyed by adapter. -#: Populated at adapter import, and consulted only to validate a table's +#: Populated at adapter import, and read only to validate a table's #: *keys* -- which happens the first time that adapter resolves a setting, by #: which point it is necessarily imported. _REGISTRY: dict[str, type[BaseConfiguration]] = {} @@ -539,7 +537,7 @@ def _register(cls: type[BaseConfiguration]) -> None: The roster in :data:`ADAPTERS` and the class are the two halves of one declaration, and this is where they are checked to agree: a class naming an adapter the roster does not list would be a configuration no file table and - no report could ever reach. + no report could ever refer to. """ adapter = cls.adapter if adapter is None or adapter not in ADAPTERS: @@ -571,8 +569,8 @@ def _env_label(env_var: str) -> str: def _toml_parser() -> Any: """The TOML parser, imported on first use. - ``import dataretrieval`` imports this module, but the parser is reachable - only once a configuration file actually exists -- the minority case. + ``import dataretrieval`` imports this module, but the parser is needed only once a + configuration file exists -- the minority case. """ if sys.version_info >= (3, 11): import tomllib @@ -582,12 +580,12 @@ def _toml_parser() -> Any: def config_path() -> Path: - """Path to the configuration file, honoring ``DATARETRIEVAL_CONFIG``. + """Path to the configuration file, applying ``DATARETRIEVAL_CONFIG``. - Memoized on the raw ``DATARETRIEVAL_CONFIG`` value, because this sits on + Memoized on the raw ``DATARETRIEVAL_CONFIG`` value, because this is on the per-request path via :func:`api_key`. Returning a stable object also lets :func:`_load_file` check its cache by identity instead of - re-normalizing a fresh ``Path``. + re-normalizing a new ``Path``. Returns ------- @@ -598,16 +596,16 @@ def config_path() -> Path: global _path_cache override = os.environ.get(CONFIG_PATH_ENV) - # Probe the memo before doing any work: this runs once per request via + # Check the memo before doing any work: this runs once per request via # ``api_key()``, so the hit path should be a dict lookup and a compare. cached = _path_cache if cached is not None and cached[0] == override: cached_guard, path = cached[1], cached[2] # The memo is only valid while whatever the path was *derived from* is # unchanged, so each branch records its own guard. A relative override - # is anchored to the working directory (a later ``os.chdir`` in a + # depends on the working directory (a later ``os.chdir`` in a # per-job notebook or scheduler must not keep reading the previous - # job's file); the default branch is anchored to ``$HOME``. An absolute + # job's file); the default branch is depends on ``$HOME``. An absolute # override depends on neither and guards with ``None``. if cached_guard is None or cached_guard == _path_guard(cached_guard[0]): return path @@ -652,8 +650,8 @@ def _resolve_against_cwd(relative: Path) -> Path: """Resolve a relative override, or report a working directory that is gone. A scratch-dir job that removes its own cwd cannot resolve a relative - ``DATARETRIEVAL_CONFIG`` at all. That surfaces as a :class:`ConfigurationError` - rather than a bare ``OSError`` escaping onto the request path -- the + ``DATARETRIEVAL_CONFIG`` at all. That is raised as a :class:`ConfigurationError` + rather than an unwrapped ``OSError`` propagating to the request path -- the taxonomy contract the rest of this module keeps. """ try: @@ -697,11 +695,11 @@ def _home_id() -> str: the path. Which variable that is differs by platform, and the memo has to agree with - the resolver or it watches a different variable. ``posixpath.expanduser`` + the resolver or it checks a different variable. ``posixpath.expanduser`` reads ``HOME``; ``ntpath.expanduser`` reads ``USERPROFILE`` (then ``HOMEDRIVE``/``HOMEPATH``) and ignores ``HOME`` outright. Preferring ``HOME`` everywhere means that on Windows -- where Git Bash and MSYS do set - it -- the memo invalidates on a variable that cannot move the path, and + it -- the memo invalidates on a variable that cannot change the path, and misses the ``USERPROFILE`` change that can. """ if os.name == "nt": @@ -715,7 +713,7 @@ def _home_id() -> str: # --- value grammar ------------------------------------------------------- # -# One parser drives each setting's grammar, so a value means the same thing and +# One parser defines each setting's grammar, so a value means the same thing and # reports the same way whichever source wrote it. Source-level adapters retain # TOML types and reject Python API type errors before producing raw strings. @@ -750,8 +748,8 @@ def _coerce_concurrency(value: object, label: str, optional: str) -> str: def _coerce_seconds(value: object, label: str, optional: str) -> str: - # Seconds, so a fractional value is meaningful -- unlike the counts, which - # are whole by nature. + # Seconds, so a fractional value is meaningful -- unlike the counts, which are + # integers. if isinstance(value, bool) or not isinstance(value, (Integral, float)): raise _type_error(label, "a number of seconds" + optional, value) return str(value) @@ -765,12 +763,12 @@ def _coerce_count(value: object, label: str, optional: str) -> str: #: Each setting's source-level type policy -- one row per setting, like #: :data:`_VALIDATORS` holds its grammar. A roster with the completeness guard -#: below rather than an if/elif chain with an implicit integer fallback, so a -#: new setting must declare its type here or fail at import -- not silently -#: parse as an integer from the typed surfaces while the untyped environment -#: accepts it. (Integers are matched as :class:`numbers.Integral` -- a numpy -#: or pandas integer is a legitimate count from Python, and ``tomllib`` only -#: ever yields ``int``, so the wider check cannot change a TOML outcome.) +#: below rather than an if/elif chain with an implicit integer fallback, so a new +#: setting must declare its type here or fail at import -- rather than parse as an +#: integer without any error from the typed surfaces while the untyped environment +#: accepts it. (Integers are matched as :class:`numbers.Integral` -- a numpy or pandas +#: integer is a legitimate count from Python, and ``tomllib`` only ever yields ``int``, +#: so the wider check cannot change a TOML outcome.) _TYPES: dict[str, Callable[[object, str, str], str]] = { "api_key": _coerce_string, "base_url": _coerce_string, @@ -796,8 +794,8 @@ def _coerce_typed(name: str, value: object, label: str, *, optional: str = "") - Shared by the two *typed* surfaces -- a configuration's fields and TOML scalars -- so a value accepted from one is accepted from the other and a - tightened rule cannot land on only half of them. (The environment is not - typed: it delivers strings, which go straight to :func:`_validate_raw`.) + tightened rule cannot apply to only half of them. (The environment is not + typed: it supplies strings, which are passed directly to :func:`_validate_raw`.) ``optional`` is the only thing that differs between them: the Python surface accepts ``None`` and says so in its messages. @@ -853,7 +851,7 @@ def _parse_seconds(raw: str, label: str) -> float: """Parse a non-negative duration in seconds; blank falls through. Seconds rather than a count, so fractional values are accepted. ``0`` - disables the bound it guards, which is why the floor is zero rather than + disables the bound it sets, which is why the minimum is zero rather than one. """ value = raw.strip() @@ -864,10 +862,10 @@ def _parse_seconds(raw: str, label: str) -> float: parsed = float(value) except ValueError as exc: raise ConfigurationError(f"{label} must be {expected} (got {raw!r}).") from exc - # ``inf`` and ``nan`` both parse as floats and both defeat the bound they - # are meant to set: ``inf`` makes every wait allowed, and ``nan`` compares - # false against every threshold. TOML has literal ``inf``/``nan``, so this - # is reachable from the file as well as from Python. + # ``inf`` and ``nan`` both parse as floats and both disable the bound they are meant + # to set: ``inf`` makes every wait allowed, and ``nan`` compares false against every + # threshold. TOML has literal ``inf``/``nan``, so this is reachable from the file as + # well as from Python. if not math.isfinite(parsed) or parsed < 0: raise ConfigurationError(f"{label} must be {expected} (got {parsed}).") return parsed @@ -890,7 +888,7 @@ def _parse_base_url(raw: str, label: str) -> str: Only the scheme is checked, and deliberately so. This module cannot know what a given service's paths look like, but it can refuse the shapes that - are never a base URL and would fail far from here -- a bare hostname that + are never a base URL and would fail later, in the request -- a bare hostname that ``httpx`` would reject, or a ``file://`` that is not a service at all. """ value = raw.strip() @@ -917,7 +915,7 @@ def _parse_progress(raw: str, label: str, *, strict: bool) -> bool: # Each integer setting's grammar, named once. The accessor and the eager -# block/TOML validator below both spell the parser this way, so a change to a +# block/TOML validator below both name the parser this way, so a change to a # bound (say ``minimum``) cannot leave a ``configure()`` block validating # against different rules than the value it later resolves. _parse_retries = partial(_parse_int, default=DEFAULT_RETRIES, minimum=0) @@ -950,7 +948,7 @@ def _named_profiles(parsed: _ParsedFile, adapter: str) -> dict[str, dict[str, An parses as a sub-table of ``[waterdata]``, and everything else in that table is a setting of the adapter's default profile. The two readers of that rule -- selecting a profile and reporting which ones exist -- share this one - definition so they cannot come to disagree about what a profile is. + definition so they cannot disagree about what a profile is. Tables are returned raw, since an adapter this process has not imported has no vocabulary to check them against. That is enough to *name* a profile, @@ -969,12 +967,11 @@ def _named_profile( ) -> dict[str, Any]: """The ``[.]`` table, checked against *allowed*. - Returns the TOML scalars as written rather than raw strings, because the - caller is :meth:`BaseConfiguration.load`, which feeds them straight back - into the configuration's own typed fields. Values are still checked here, - with a label that names the file and the table: a grammar error found on - the way *out* of the file should say which line to fix, not merely which - field of which class ended up holding it. + Returns the TOML scalars as written rather than raw strings, because the caller is + :meth:`BaseConfiguration.load`, which passes them directly to the configuration's + own typed fields. Values are still checked here, with a label that names the file + and the table: a grammar error found when reading the file should say which line to + fix, not only which field of which class ended up holding it. """ path, parsed = _current_file() named = _named_profiles(parsed, adapter) @@ -994,9 +991,9 @@ def _named_profile( table = named[profile] # A profile is one flat set of settings for one adapter, so a table inside - # one is a shape the grammar has no reading for -- most likely a file + # one is a shape the grammar does not define -- most likely a file # migrated from the retired ``[profiles.bulk.ngwmn]``, where a profile did - # carry per-service detail. Dropping it silently would leave the author + # contain per-service detail. Dropping it without an error would leave the author # believing they had tuned something. Checked here rather than at parse # time for the same reason keys are: a malformed profile for one adapter # must not fail another adapter's call. @@ -1018,9 +1015,9 @@ def _named_profile( def _current_file() -> tuple[Path, _ParsedFile]: """The config file as currently loaded: its path and its parsed form. - One helper so the two always travel together. They are a single fact, and - handing the top-level scope a different ``_ParsedFile`` than the - adapter scope saw in the same resolution is exactly the drift that made an + One helper so the two are always read together. They are a single fact, and + giving the top-level scope a different ``_ParsedFile`` than the + adapter scope in the same resolution is the drift that made an adapter-scoped read load the file twice. """ path = config_path() @@ -1038,7 +1035,7 @@ def _adapter_file_settings( caller selects one, so they are skipped here (see :func:`_accepted_keys`). Validated on first use, not at parse time, so an invalid value in ``[nldi]`` - cannot fail a Water Data call -- the blast-radius rule ADR 0010 set. + cannot fail a Water Data call -- the isolation rule ADR 0010 set. """ table = parsed.adapters.get(adapter) if not table: @@ -1076,12 +1073,12 @@ def _load_file(path: Path) -> _ParsedFile: f"configuration path {path} is a directory, not a file." ) - # Only a regular file is parsed. Anything else readable -- a character - # device, a FIFO -- is treated as *empty* configuration without being - # opened, which is what ``DATARETRIEVAL_CONFIG=/dev/null`` asks for and the - # only coherent answer for a stream: settings are re-resolved on every - # request, so a FIFO would hand its contents to the first getter and - # nothing to the rest, making the API key vanish mid-run. + # Only a regular file is parsed. Anything else readable -- a character device, a + # FIFO -- is treated as *empty* configuration without being opened, which is what + # ``DATARETRIEVAL_CONFIG=/dev/null`` means and the only coherent behavior for a + # stream: settings are re-resolved on every request, so a FIFO would deliver its + # contents only to the first getter and nothing to the rest, so the API key would be + # missing mid-run. if not stat.S_ISREG(st.st_mode): return _ParsedFile(exists=True) @@ -1109,13 +1106,12 @@ def _stat_config_file(path: Path) -> os.stat_result | None: def _cached_parse_by_metadata(path: Path, st: os.stat_result) -> _ParsedFile | None: """The cached parse when the metadata stamp still matches, else ``None``. - POSIX ``st_ctime_ns`` advances on any inode change, so the metadata stamp - catches even a rewrite that restores the original mtime. Windows ctime is - *creation* time, so there the stamp cannot see that class of edit and the - content compare in :func:`_parse_or_reuse_cache` is the only check that - catches it -- the re-read it forces is deliberate, and - ``test_file_edit_is_picked_up`` pins it. Do not drop the ctime gate (or - extend the stamp to Windows) without a Windows-safe change detector. + POSIX ``st_ctime_ns`` advances on any inode change, so the metadata stamp detects + even a rewrite that restores the original mtime. Windows ctime is *creation* time, + so there the stamp cannot detect that class of edit and the content compare in + :func:`_parse_or_reuse_cache` is the only check that detects it -- the re-read it + forces is deliberate, and ``test_file_edit_is_picked_up`` pins it. Do not drop the + ctime gate (or extend the stamp to Windows) without a Windows-safe change detector. """ cached = _file_cache if ( @@ -1172,9 +1168,9 @@ def _interpret(data: dict[str, Any], path: Path) -> _ParsedFile: Only the top-level table is validated here, because it always applies. An adapter's table is kept raw and validated when that adapter first resolves a setting: an invalid value in ``[nldi]`` must not fail a Water Data call, - the same blast-radius rule :func:`~dataretrieval.utils._default_headers` - follows for the key itself. It is also what lets an adapter's vocabulary - live in the adapter, which this module cannot import. + the same isolation rule :func:`~dataretrieval.utils._default_headers` + applies to the key itself. It is also what lets an adapter's vocabulary + be defined in the adapter, which this module cannot import. """ top: dict[str, Any] = {} adapters: dict[str, dict[str, Any]] = {} @@ -1191,8 +1187,8 @@ def _interpret(data: dict[str, Any], path: Path) -> _ParsedFile: if key == _RETIRED_PROFILES_TABLE: # A file written against the earlier design, where one profile # switched every service at once. The generic message below would - # send its author hunting for a typo in a table that is spelled - # exactly as the old docs said, so name the replacement instead. + # leave its author looking for a typo in a table written + # as the old docs specified, so name the replacement instead. raise ConfigurationError( f"{path}: [{_RETIRED_PROFILES_TABLE}] is no longer read. A " "profile now belongs to one adapter: write [.] " @@ -1218,11 +1214,11 @@ def _accepted_keys( ) -> dict[str, Any]: """Filter one table down to the settings it is allowed to name. - The key policy for every table in the file, in one place, so the default - profile and a named profile cannot come to disagree about what is a typo. - An unrecognized name warns rather than raising, so a file written for a - newer release still works; a name this release *does* know but that table - cannot use raises, because that one can never become meaningful. + The key policy for every table in the file, in one place, so the default profile and + a named profile cannot disagree about what is a typo. An unrecognized name warns + rather than raising, so a file written for a newer release still works; a name this + release recognizes but that table cannot use raises, because that name can never + become meaningful. """ out: dict[str, Any] = {} for key, value in table.items(): @@ -1230,14 +1226,14 @@ def _accepted_keys( # A named profile -- ``[waterdata.bulk]`` parses as a sub-table of # ``[waterdata]``. Inert until a caller selects it, so it is # neither a setting here nor an error. Only an adapter's table can - # reach this: the top level rejects unknown tables when it parses, + # be passed here: the top level rejects unknown tables when it parses, # and :func:`_named_profile` refuses a table inside a profile, so a # sub-table here is always a profile rather than deeper nesting. continue if key in ADAPTER_ONLY_SETTINGS: - # Rejected from the file wherever it appears. A file that silently + # Rejected from the file wherever it appears. A file that # redirects a data-retrieval library to another host is a - # supply-chain-shaped hazard; an in-code block keeps the redirect + # supply-chain hazard; an in-code block keeps the redirect # where a reader of the script sees it (ADR 0011). raise ConfigurationError( f"{path}: {key!r} at {where} may only be set in code, in a " @@ -1247,7 +1243,7 @@ def _accepted_keys( if key in SETTINGS: # A real setting, in a table that does not read it. Unlike an # unrecognized name -- which may belong to a newer release -- - # this cannot become meaningful later, and silently ignoring it + # this cannot become meaningful later, and ignoring it without an error # would leave a caller believing they had tuned something. See # ADR 0010. raise ConfigurationError( @@ -1273,14 +1269,12 @@ def _checked_table( ) -> dict[str, tuple[Any, str]]: """Check one table of the file, in both the forms its two readers need. - Every table in the file comes through here: the top-level keys, an - adapter's default profile, and a named profile. They differ only in what - they do with the result -- the chain wants raw strings, a profile being - loaded wants the TOML scalars to hand back to a configuration's own typed - fields -- so both are returned and each reader takes its half. Written once - because the checks are the interesting part and they must not diverge: a - per-table policy added for one kind of table would otherwise skip the - other, silently. + Every table in the file comes through here: the top-level keys, an adapter's default + profile, and a named profile. They differ only in what they do with the result -- + the chain takes raw strings, a profile being loaded takes the TOML scalars to pass + to a configuration's own typed fields -- so both are returned and each reader takes + its half. Written once because the checks are the part that must not diverge: a + per-table policy added for one kind of table would otherwise skip the other. ``tomllib`` returns typed scalars (``concurrency = 32`` is an ``int``, ``concurrency = "unbounded"`` a ``str``), so types are checked here before @@ -1323,7 +1317,7 @@ def _holds_api_key(parsed: _ParsedFile) -> bool: """Whether the file names an API key anywhere, including inert tables. Inert tables count because the question is what the *file* contains, not - what this run resolves: a key sitting in a profile nobody selected is just + what this run resolves: a key in a profile nobody selected is just as readable to another user on the machine. """ if "api_key" in parsed.base: @@ -1342,7 +1336,7 @@ def _warn_on_loose_permissions( Follows the ``~/.ssh`` and ``.netrc`` convention, but warns rather than refusing -- shared filesystems on HPC clusters have their own conventions, - and refusing to read would strand those users. + and refusing to read would leave those users unable to use the file. """ if os.name != "posix" or path in _permission_warned: return diff --git a/dataretrieval/_deprecation.py b/dataretrieval/_deprecation.py index de486f920..4c8f19683 100644 --- a/dataretrieval/_deprecation.py +++ b/dataretrieval/_deprecation.py @@ -1,8 +1,8 @@ """One advisory mechanism, and one place to read the removal horizons. Every deprecation is announced through this module, with a horizon in -:data:`REMOVALS` (ADR 0012). A ``DeprecationWarning`` promises that a *name in -this package* is going away, while an advisory that an upstream *dataset* has +:data:`REMOVALS` (ADR 0012). A ``DeprecationWarning`` states that a *name in +this package* is being removed, while an advisory that an upstream *dataset* has stopped being updated belongs under :class:`~dataretrieval.exceptions.DataCurrencyWarning` (ADR 0004). """ @@ -13,7 +13,7 @@ #: Published removal horizons, by the surface each covers. A date here is a #: commitment already made in a released warning message; read it rather than -#: spelling a date at the call site, so bumping one is a single edit. +#: spelling a date at the call site, so extending one is a single edit. REMOVALS: dict[str, str] = { "nwis": "2027-05-06", "waterdata.get_cql(service=)": "2027-08-09", @@ -35,19 +35,19 @@ def warn_deprecated( Parameters ---------- subject - What is going away, as the caller spells it (``"nwis.get_dv"``, the + What is being removed, as the caller spells it (``"nwis.get_dv"``, the keyword ``"stateFips"``). replacement - What to use instead. Named in every message because a deprecation - without a migration path is only an inconvenience. + What to use instead. Named in every message because a deprecation without a + replacement gives the caller nothing to do. removal Date from :data:`REMOVALS`, or ``None`` when no horizon has been - published -- which reads as "a future release" rather than inventing - a commitment. + published -- which reads as "a future release" rather than stating + a date that has not been decided. detail - Optional sentence appended after the advisory, for a rename whose - reason is worth giving. Appended, never interpolated into the - message, so a multi-sentence detail cannot corrupt the wording. + Optional sentence appended after the advisory, for a rename whose reason should + be stated. Appended, never interpolated into the message, so a multi-sentence + detail cannot corrupt the wording. stacklevel Frames to skip so the warning is attributed to the caller's own line, not to this function. diff --git a/dataretrieval/_querying.py b/dataretrieval/_querying.py index aa360a122..ad1c2c803 100644 --- a/dataretrieval/_querying.py +++ b/dataretrieval/_querying.py @@ -1,17 +1,17 @@ """The one-shot HTTP query path behind the legacy service adapters. "Compose a USGS query URL, send it, map the status, retry a transient" -- the -half of the old ``utils`` module that talks to the network, as used by ``nwis``, +half of the old ``utils`` module that issues requests, as used by ``nwis``, ``wqp``, ``nldi``, ``streamstats`` and ``nwdc``. Its other half (pandas -column munging) shared nothing with this but a filename: no caller wanted both, +column reshaping) shared nothing with this but a filename: no caller used both, and the two have disjoint dependencies -- this one needs ``exceptions`` and ``transport``, that one needs ``codes`` and pandas. The module is private because the *names* are not: ``query`` and ``to_str`` keep their documented ``dataretrieval.utils`` path, the way ``Ambient`` and ``BaseMetadata`` do from their own implementation leaves. This is legacy -machinery for the deprecated single-request adapters; new service code belongs -on the chunked transport instead. +code for the deprecated single-request adapters; new service code uses +the chunked transport instead. """ from __future__ import annotations @@ -108,11 +108,11 @@ def _raise_for_status( remediation as the client-side over-long-URL case below, rather than a bare ``HTTP 414`` (both still raise :class:`~dataretrieval.exceptions.URLTooLong`). - ``detail_from``, when given, is called *only on an error response* to pull an - API-specific detail string (e.g. a JSON error envelope's message) out of the - body; a truthy result is appended to the raised message. This lets callers - surface their API's error wording without re-implementing the status-to-type - mapping and message format. + ``detail_from``, when given, is called *only on an error response* to read an + API-specific detail string (e.g. a JSON error envelope's message) from the body; a + truthy result is appended to the raised message. This lets callers include their + API's error wording without re-implementing the status-to-type mapping and message + format. """ status = response.status_code if status < 400: @@ -134,9 +134,9 @@ def _raise_for_status( def _single_request_policy(adapter: str | None = None) -> RetryPolicy: """Retry policy for the one-shot adapters (WQP, NLDI, StreamStats). - These services answer a rejected query with a 500, so only the gateway - statuses are worth re-sending; the Water Data chunker keeps the broader - default, where a 5xx is a transient upstream failure worth riding out. + These services respond to a rejected query with a 500, so only the gateway statuses + are re-sent; the Water Data chunker keeps the broader default, where a 5xx is a + transient upstream failure that a later attempt may not see. ``adapter`` names which settings table supplies ``retries`` and ``stall_timeout`` -- these three services share a retry *shape* but not @@ -215,8 +215,8 @@ def _query_with_retry( **HTTPX_DEFAULTS, ) - # USGS waterservices signals an empty result with a 200 whose body starts - # "No sites/data ..." (its legacy wording); surface it as NoSitesError. + # USGS waterservices reports an empty result with a 200 whose body starts + # "No sites/data ..." (its legacy wording); raise it as NoSitesError. if response.text.startswith("No sites/data"): raise NoSitesError(response.url) diff --git a/dataretrieval/_response_metadata.py b/dataretrieval/_response_metadata.py index e19321ff0..6df404921 100644 --- a/dataretrieval/_response_metadata.py +++ b/dataretrieval/_response_metadata.py @@ -2,7 +2,7 @@ A dependency-free leaf on purpose (ADR 0003). This class is the second half of the ``(DataFrame, metadata)`` return contract (ADR 0007), so nearly every -service module needs it; importing it pulls in nothing but ``httpx``. +service module needs it; importing it imports nothing but ``httpx``. ``dataretrieval.utils.BaseMetadata`` remains the public import. """ @@ -29,7 +29,7 @@ class BaseMetadata: """ def __init__(self, response: httpx.Response) -> None: - """Generate a standard set of metadata informed by the response. + """Generate a standard set of metadata from the response. Parameters ---------- @@ -53,7 +53,7 @@ def __init__(self, response: httpx.Response) -> None: @property def site_info(self) -> Any: raise NotImplementedError( - "This metadata object carries no site_info: only the nwis and wqp " + "This metadata object has no site_info: only the nwis and wqp " "metadata classes implement it, and the getter that produced this " "result does not return site descriptions alongside data. Fetch " "them using a separate function from the same adapter -- " diff --git a/dataretrieval/_validation.py b/dataretrieval/_validation.py index b11fb1869..09ded82d0 100644 --- a/dataretrieval/_validation.py +++ b/dataretrieval/_validation.py @@ -6,7 +6,7 @@ alternatives, and mutually exclusive arguments. Every check raises ``ValueError`` about an argument's *value*, so calling code -needs no inventory of which check fired. Every message states the problem and +needs no inventory of which check raised. Every message states the problem and an executable correction. """ @@ -47,7 +47,7 @@ def _qualify(context: str, *, prefix: str = " ") -> str: """Return *context* ready to splice into a message, or nothing. Every check appends its caller's ``context`` the same way; owning the - splice here keeps a new check from inventing a fifth local spelling of + splice here keeps a new check from adding a fifth local copy of ``f" {context}" if context else ""``. """ return f"{prefix}{context}" if context else "" @@ -59,7 +59,7 @@ def _supplied(values: Mapping[str, object]) -> tuple[list[str], list[str]]: ``None`` is the package's "not supplied" marker throughout the public signatures, so it is the one this module tests for. A caller whose sentinel differs -- an empty string that should count as missing -- normalizes to - ``None`` before calling, rather than this module guessing which falsy values + ``None`` before calling, rather than this module deciding which falsy values were meant. """ supplied: list[str] = [] @@ -91,14 +91,14 @@ def require_one_of( name What the value *is*, as the caller's parameter names it (``"service"``, ``"collection"``). It becomes the message's subject, so it must match - the parameter the caller actually passed. + the parameter the caller passed. context Optional qualifier for a vocabulary that depends on another argument, e.g. ``context="service 'wqp'"`` when the valid profiles differ per service. remedy - A further move, for a vocabulary narrower than the service's: how to - reach what this function does not accept. Added rather than + A further step, for a vocabulary narrower than the service's: how to + obtain what this function does not accept. Added rather than substituted -- unlike the checks below, there is no derived remedy here, since naming the options *is* the message. @@ -109,7 +109,7 @@ def require_one_of( """ if isinstance(options, str): # ``str`` is a Collection, so this type-checks -- and then ``in`` - # silently means "substring", accepting any fragment of a valid option. + # means "substring" without any error, accepting any fragment of a valid option. raise TypeError(f"options must be a collection of values, not {options!r}") if value in options: return @@ -131,8 +131,8 @@ def require_argument( """Return *value*, or raise ``ValueError`` if it was not supplied. Returns the value rather than ``None`` so the check also narrows the type: - a caller that must hand an optional argument to something requiring a - concrete one writes ``x = require_argument("x", x)`` and is done. The + a caller that must pass an optional argument to something requiring a + concrete one writes ``x = require_argument("x", x)``. The alternative -- validating here and re-testing for ``None`` to satisfy the type checker -- puts a second, unreachable message next to this one, and the two drift. @@ -174,10 +174,10 @@ def require_together( ) -> None: """Raise ``ValueError`` unless *values* are all supplied or all omitted. - For arguments that only mean something as a set -- a ``lat``/``long`` pair, - a ``feature_source``/``feature_id`` pair. Passing none of them is allowed: - that is the caller declining the whole group, which is a different question - from whether the group is complete. + For arguments that only mean something as a set -- a ``lat``/``long`` pair, a + ``feature_source``/``feature_id`` pair. Passing none of them is allowed: that is the + caller omitting the whole group, which is a different case from whether the group is + complete. Parameters ---------- @@ -218,8 +218,8 @@ def require_any_of( """Raise ``ValueError`` unless at least one of *values* was supplied. For a query that needs to be narrowed but does not care how -- the NWIS - major filters, where any one of five is enough for the service to answer. - The permissive sibling of :func:`require_exactly_one`: two of them is a + major filters, where any one of five is enough for the service to respond. + The permissive counterpart of :func:`require_exactly_one`: two of them is a narrower query rather than a contradiction, so only none is an error. ``None`` counts as not supplied, so ``sites=None`` is refused rather than reaching the URL. @@ -258,14 +258,13 @@ def require_exactly_one( ) -> tuple[str, _T]: """Return the one supplied ``(name, value)``, or raise ``ValueError``. - For a choice between alternatives that are each sufficient on their own -- - the origin of an NLDI navigation, the location selector of an NWDC query. - Both failure directions are reported by the same check because they have - the same fix from opposite sides: supply one, or drop the rest. The - winning pair is returned for the same reason :func:`require_argument` - returns its value: the caller's next move is to dispatch on it, and - re-deriving it beside the call restates the invariant this check just - proved. + For a choice between alternatives that are each sufficient on their own -- the + origin of an NLDI navigation, the location selector of an NWDC query. Both failure + directions are reported by the same check because they have the same fix in either + direction: supply one, or drop the rest. The selected pair is returned for the same + reason :func:`require_argument` returns its value: the caller's next step is to + dispatch on it, and re-deriving it beside the call restates the invariant this check + established. Parameters ---------- @@ -313,7 +312,7 @@ def reject_together( ) -> None: """Raise ``ValueError`` if more than one of *values* was supplied. - The permissive sibling of :func:`require_exactly_one`: it rejects the + The permissive counterpart of :func:`require_exactly_one`: it rejects the combination without requiring that anything be supplied at all, for arguments that conflict but are jointly optional. diff --git a/dataretrieval/_wqx.py b/dataretrieval/_wqx.py index 733a1c5cb..3d1312a14 100644 --- a/dataretrieval/_wqx.py +++ b/dataretrieval/_wqx.py @@ -2,8 +2,8 @@ The Samples database and the Water Quality Portal both split an instant across three columns -- a date, a time, and a time-zone abbreviation -- and they spell -the trio two different ways. Recognizing either spelling and folding it into one -UTC column is service-specific knowledge, so it lives in its own leaf rather +the trio two different ways. Recognizing either spelling and combining it into one +UTC column is service-specific knowledge, so it is in its own leaf rather than in :mod:`dataretrieval.utils` (ADR 0001). Depends on pandas and the time-zone table only; nothing here issues a request. @@ -106,11 +106,10 @@ def _attach_datetime_columns(df: pd.DataFrame) -> pd.DataFrame: are left intact, and an existing ``DateTime`` column is never overwritten. - Rows are sorted (and the index reset) by the canonical activity-start - datetime when present — ``Activity_StartDateTime`` (WQX3) or - ``ActivityStartDateTime`` (legacy WQP) — falling back to the first - detected ``*Date`` column. Mirrors R ``dataRetrieval``'s - end-of-pipeline sort in ``importWQP.R``. + Rows are sorted (and the index reset) by the canonical activity-start datetime when + present — ``Activity_StartDateTime`` (WQX3) or ``ActivityStartDateTime`` (legacy + WQP) — falling back to the first detected ``*Date`` column. Matches R + ``dataRetrieval``'s end-of-pipeline sort in ``importWQP.R``. Parameters ---------- diff --git a/dataretrieval/codes/states.py b/dataretrieval/codes/states.py index caa1cb747..e0d828d21 100644 --- a/dataretrieval/codes/states.py +++ b/dataretrieval/codes/states.py @@ -164,8 +164,8 @@ def to_state( Coverage is the 50 states, DC, and the five US territories, each under its real ANSI/FIPS code. A ``value`` that isn't recognized in one of those - encodings raises ``ValueError``, so a typo fails fast rather than - silently matching nothing. + encodings raises ``ValueError``, so a typo raises rather than + matching nothing. """ if isinstance(value, str): return _to_state_one(value, to) @@ -224,16 +224,15 @@ def apply_state( ``state`` alongside any of them raises ``ValueError``. Returns the (mutated) ``local_vars``. - An unrecognized ``state`` is re-raised naming the parameters in ``reject``, - and only those. They are the endpoint's own state parameters as the public - getter spells them: the mutual-exclusion guard below proves the getter - accepts them as keyword arguments. ``into`` is deliberately not offered, - because an API query parameter need not exist on the getter's signature -- - NGWMN's ``get_sites`` filters on ``state_name`` but accepts only ``state``, - so naming ``into`` there produced a remedy that raises ``TypeError`` when - followed. An endpoint with an empty ``reject`` has no alternative spelling - to offer, so it appends nothing rather than pointing back at the argument - that just failed. + An unrecognized ``state`` is re-raised naming the parameters in ``reject``, and only + those. They are the endpoint's own state parameters as the public getter spells + them: the mutual-exclusion check below shows the getter accepts them as keyword + arguments. ``into`` is deliberately not offered, because an API query parameter need + not exist on the getter's signature -- NGWMN's ``get_sites`` filters on + ``state_name`` but accepts only ``state``, so naming ``into`` there produced a + remedy that raises ``TypeError`` when followed. An endpoint with an empty ``reject`` + has no alternative spelling to offer, so it appends nothing rather than naming the + argument that just failed. """ state = local_vars.pop("state", None) if state is None: @@ -249,7 +248,7 @@ def apply_state( # No native spelling of the getter's own to offer instead. raise # Only ``reject`` proves the getter accepts a spelling; ``into`` is the - # API query parameter, so it leads only when it appears there too. + # API query parameter, so it is listed only when it appears there too. offered = dict.fromkeys(n for n in (into, *reject) if n in reject) raise ValueError( f"{err} Pass {' or '.join(offered)} directly instead, using the " diff --git a/dataretrieval/combining.py b/dataretrieval/combining.py index 167c1814b..ae0dec2d7 100644 --- a/dataretrieval/combining.py +++ b/dataretrieval/combining.py @@ -19,7 +19,7 @@ import httpx import pandas as pd -# Response header USGS uses to advertise remaining hourly quota. Lives in this +# Response header USGS uses to report remaining hourly quota. Defined in this # module so every layer (the combine helpers below, the engine's per-page # progress reporter) reads it from one place rather than hard-coding the string. _QUOTA_HEADER = "x-ratelimit-remaining" @@ -46,16 +46,16 @@ def _safe_elapsed(response: httpx.Response) -> timedelta: def _set_response_url(response: httpx.Response, url: str | httpx.URL) -> None: """ - Overwrite the URL surfaced by a response without back-propagating + Overwrite the URL a response reports without back-propagating the change into any aliased original. - Lightweight test doubles expose ``.url`` as a writable attribute. Real - :class:`httpx.Response` objects resolve it through a bound request, so swap - in a fresh request carrying the new URL; mutating the existing request would - leak through any shallow copy that shares it. + Minimal test doubles expose ``.url`` as a writable attribute. Real + :class:`httpx.Response` objects resolve it through a bound request, so attach a new + request with the new URL; mutating the existing request would leak through any + shallow copy that shares it. """ if not isinstance(response, httpx.Response): - # Lightweight test doubles expose ``url`` as a writable attribute. + # Minimal test doubles expose ``url`` as a writable attribute. response.url = url return @@ -74,7 +74,7 @@ def _lowest_remaining(responses: list[httpx.Response]) -> httpx.Response: """The response reporting the lowest ``x-ratelimit-remaining``. Within a rate-limit window, the counter decreases monotonically, so the - smallest value observed is the most conservative value to surface. Under + smallest value observed is the most conservative value to report. Under concurrent fan-out, the last response *by index* need not be the one the server processed last. Fall back to the last response when none reports the header. @@ -98,14 +98,14 @@ def _merge_response( elapsed: timedelta, url: str | httpx.URL | None = None, ) -> httpx.Response: - """Fold several responses into one shallow copy of ``base``. + """Merge several responses into one shallow copy of ``base``. - The copy's ``.headers`` are rebuilt as a fresh ``httpx.Headers`` from + The copy's ``.headers`` are rebuilt as a new ``httpx.Headers`` from ``headers_from``, ``.elapsed`` is set to ``elapsed``, and ``.url`` is overridden when ``url`` is given. ``base`` and ``headers_from`` are never - mutated, and the fresh ``httpx.Headers`` means downstream mutations don't - back-propagate into any underlying response — so callers may re-fold - idempotently. This is the one low-level merge behind both pagination + mutated, and the new ``httpx.Headers`` means downstream mutations don't + back-propagate into any underlying response — so callers may re-merge + idempotently. This is the one low-level merge used by both pagination (:func:`~dataretrieval.transport.pagination.paginate`) and the chunked / fan-out aggregation (:func:`_combine_chunk_responses`).""" merged = copy.copy(base) @@ -120,7 +120,7 @@ def _combine_chunk_frames(frames: list[pd.DataFrame]) -> pd.DataFrame: """Concatenate per-chunk frames and deduplicate IDs across chunks. Empty frames are ignored before concatenation so an empty plain - :class:`pandas.DataFrame` cannot downgrade a real ``GeoDataFrame`` and + :class:`pandas.DataFrame` cannot reduce a ``GeoDataFrame`` to a plain frame and strip its geometry or CRS. When every frame is empty, the first frame is returned to preserve its concrete type. @@ -128,8 +128,8 @@ def _combine_chunk_frames(frames: list[pd.DataFrame]) -> pd.DataFrame: deduplicated regardless of the plan axis. Filter clauses can match the same feature, and list inputs can contain repeated values or otherwise select overlapping records. Rows without an ``id`` are preserved verbatim: pandas - treats null values as duplicates, so deduplicating them would silently lose - data. + treats null values as duplicates, so deduplicating them would drop rows with no + indication. """ non_empty = [frame for frame in frames if not frame.empty] if not non_empty: @@ -155,7 +155,7 @@ def _combine_chunk_responses( responses: list[httpx.Response], canonical_url: str | None ) -> httpx.Response: """ - Fold per-chunk responses into a single aggregated response. + Merge per-chunk responses into a single aggregated response. For a multi-response input, returns a shallow copy of ``responses[0]`` with ``.headers`` set to those of the response reporting @@ -176,7 +176,7 @@ def _combine_chunk_responses( canonical_url : str or None URL of the unchunked original request. ``None`` skips the URL override — used by the passthrough path (the fetcher's - response already carries the original-query URL) and by the + response already has the original-query URL) and by the worst-case overflow path (no buildable canonical URL exists). Returns @@ -188,17 +188,16 @@ def _combine_chunk_responses( ``url`` are never mutated), so it's safe to call repeatedly via :attr:`ChunkedCall.partial_response` during error inspection or resume retries. ``headers`` on the returned - object is a fresh ``httpx.Headers``, so mutations there don't + object is a new ``httpx.Headers``, so mutations there don't back-propagate into any chunk's underlying response. """ if len(responses) == 1 and canonical_url is None: return responses[0] # Headers come from the response with the lowest reported remaining quota - # (``_lowest_remaining`` returns the lone response as-is for a - # single-element list). ``_merge_response`` re-sums elapsed onto a - # fresh copy, so repeated calls (e.g. via ``ChunkedCall.partial_response`` - # during resume) stay idempotent. + # (``_lowest_remaining`` returns the lone response as-is for a single-element list). + # ``_merge_response`` re-sums elapsed onto a new copy, so repeated calls (e.g. via + # ``ChunkedCall.partial_response`` during resume) stay idempotent. elapsed = sum((_safe_elapsed(r) for r in responses), start=timedelta()) return _merge_response( responses[0], diff --git a/dataretrieval/configuration.py b/dataretrieval/configuration.py index 0fb951308..3ba76966b 100644 --- a/dataretrieval/configuration.py +++ b/dataretrieval/configuration.py @@ -1,6 +1,6 @@ """Layered configuration resolution for ``dataretrieval``. -Every tunable setting -- the Water Data API key, the fan-out concurrency cap, +Every setting -- the Water Data API key, the fan-out concurrency cap, the retry count, and the progress line -- resolves through one ordered chain so a caller never has to mutate ``os.environ`` to configure a single call. @@ -26,7 +26,7 @@ Precedence applies **per setting**, not per source: an environment that sets only ``API_USGS_PAT`` leaves a file-provided ``concurrency`` fully in effect. The environment ranks above the file (ADR 0009). ADR 0011 makes one exception: a -profile named *in code* reaches the chain through :func:`configure`, above the +profile named *in code* enters the chain through :func:`configure`, above the environment. A caller configures by passing configuration objects, at most one per adapter:: @@ -140,7 +140,7 @@ def configure(*configurations: BaseConfiguration) -> Iterator[None]: """Apply configuration profiles for the duration of a ``with`` block. - The highest-precedence source. Takes configuration objects positionally, at + This is the highest-precedence source. Takes configuration objects positionally, at most one per adapter, and nothing else:: with dataretrieval.configure( @@ -151,9 +151,10 @@ def configure(*configurations: BaseConfiguration) -> Iterator[None]: df, md = waterdata.get_daily(monitoring_location_id=sites) The adapter a configuration targets is a property of its class, so the - caller never restates it -- which is what keeps the adapter roster from - being spelled once per call site. Naming two configurations for one adapter - raises: they are the one pairing with no defined order between them. + caller never restates it -- which is what keeps the caller from + restating the adapter roster at every call site. Naming two + configurations for one adapter raises: they are the one pairing with no + defined order between them. Because the block is delivered through a :class:`~contextvars.ContextVar`, a value set here applies to the current thread and to asyncio tasks started @@ -163,14 +164,14 @@ def configure(*configurations: BaseConfiguration) -> Iterator[None]: Blocks nest and merge per setting: an inner block that sets only ``concurrency`` keeps the outer block's ``api_key``, and an adapter - configuration in an outer block loses to a package-wide value set by a + configuration in an outer block is overridden by a package-wide value set by a block nested inside it, so the innermost block always decides. Parameters ---------- *configurations : BaseConfiguration A package-wide :class:`Configuration` and/or one configuration per - adapter, in any order. Each adapter's class lives in that adapter's + adapter, in any order. Each adapter's class is defined in that adapter's module -- ``WaterdataConfiguration`` in :mod:`dataretrieval.waterdata`, ``NgwmnConfiguration`` in :mod:`dataretrieval.ngwmn`, and so on. @@ -195,7 +196,7 @@ def configure(*configurations: BaseConfiguration) -> Iterator[None]: ): df, md = waterdata.get_daily(monitoring_location_id="USGS-05114000") - # a big overnight pull, from a [waterdata.bulk] table in the file + # a large overnight pull, from a [waterdata.bulk] table in the file with dataretrieval.configure(WaterdataConfiguration.load("bulk")): df, md = waterdata.get_daily(monitoring_location_id=many_sites) @@ -215,11 +216,12 @@ def _frame(configurations: tuple[BaseConfiguration, ...]) -> _Frame: back to raw strings here so that every source shares one parser and one set of error messages; they were already checked when each configuration was constructed, so nothing new can fail at this point except the two - call-shaped mistakes below. Rendering is therefore all this asks for -- + mistakes in how ``configure()`` was called, below. Rendering is therefore + all this function does -- :func:`_coerce_typed` rather than :func:`_validated_raw`, so a value is not - put through its grammar a second time on every block entry. Construction - stays the single validation point, which is where a typo should raise - anyway: at the line that wrote it, not at a later ``with`` statement. + put through its grammar a second time on every block entry. Construction stays the + single validation point, which is where a typo should raise: at the line that wrote + it, not at a later ``with`` statement. Each value is stored with the label naming the configuration it came from, because this is the last point where that is known -- see :data:`_Frame`. @@ -269,7 +271,7 @@ def _configuration_overrides( def show_configuration(*, stream: TextIO | None = None) -> None: """Print the effective configuration and where each setting came from. - A debugging aid for "why is this using my old key?". Every value is + A debugging aid for finding which source supplied a value. Every value is reported with the origin that supplied it, named exactly: which variable, which table of the file, and -- when a caller selected one -- which profile. The API key is never printed, only whether one is set. @@ -297,7 +299,7 @@ def show_configuration(*, stream: TextIO | None = None) -> None: parallel_chunks 1 built-in default stall_timeout 60s built-in default - A built-in default is package-wide. An adapter may prefer its own for + A built-in default is package-wide. An adapter may use its own default for its own calls; a value from any source above overrides both. adapter overrides @@ -315,9 +317,9 @@ def show_configuration(*, stream: TextIO | None = None) -> None: path = config_path() except ConfigurationError as exc: # Resolution itself can fail (a relative override with the working - # directory removed). That is precisely a configuration a caller would + # directory removed). That is a configuration a caller would # run this to understand, so report it as the file row rather than - # raising out of the explainer. + # raising from the report. print(f"config file ", file=out) return @@ -341,9 +343,9 @@ class _ErrorDeduplicatingCell: The report exists to explain a configuration, and the configurations most in need of explaining are the broken ones -- an unparseable file, a value that fails its grammar, a profile that no longer exists. Nothing - here raises: each distinct failure is printed once, in the first place - it shows up; a repeat is collapsed, so one invalid file does not bury the - rows that did resolve under ten copies of the same message. + here raises: each distinct failure is printed once, in the first row + it appears; a repeat is collapsed, so one invalid file does not repeat the + same message on every row that did resolve. """ def __init__(self) -> None: @@ -367,9 +369,9 @@ def __call__(self, render: Callable[[], object]) -> str: def _show_file_status( out: TextIO, path: Path, cell: _ErrorDeduplicatingCell ) -> _ParsedFile: - """Probe and print the config file status line, returning the parsed file. + """Read and print the config file status line, returning the parsed file. - Probing the file once here means a whole-file problem -- unparseable TOML, + Reading the file once here means a whole-file problem -- unparseable TOML, an invalid value at the top level -- is reported on the file row rather than repeated in every setting's row below. """ @@ -396,7 +398,7 @@ def _show_built_in_default_note(out: TextIO, rows: list[tuple[str, str, str]]) - """Print the built-in default footnote when at least one row uses it.""" if any(label == _BUILT_IN for _name, _value, label in rows): print( - "\nA built-in default is package-wide. An adapter may prefer its own " + "\nA built-in default is package-wide. An adapter may use its own default " "for\nits own calls; a value from any source above overrides both.", file=out, ) @@ -409,13 +411,13 @@ def _show_adapter_overrides( ) -> None: """Print the adapter-scoped settings that differ from the rows above. - Only settings actually overridden, and only adapters that override one: a - full adapter-by-setting grid would be mostly inherited values, burying the - answer to "what will this call use" under the rows that change nothing. + Only settings overridden, and only adapters that override one: a + full adapter-by-setting grid would be mostly inherited values, obscuring which + value a call will use among rows that change nothing. Each row names its origin exactly, which for a selected profile is the profile: ``configure() block [waterdata.bulk]`` rather than a bare block, - so the report answers *which* profile put that value there. + so the report names *which* profile supplied the value. An adapter this process has not imported has no vocabulary to resolve against, so it is skipped here and named by @@ -442,7 +444,7 @@ def _collect_adapter_overrides( """Gather adapter-scoped settings that differ from the package-wide rows. Separated from :func:`_show_adapter_overrides` so the collection logic -- - which carries the nesting -- is not interleaved with the formatting logic. + which contains the nesting -- is not interleaved with the formatting logic. """ overrides: list[tuple[str, str, str, str]] = [] for adapter in ADAPTERS: @@ -480,16 +482,16 @@ def _overrides_for_adapter( def _show_profiles(out: TextIO, parsed: _ParsedFile) -> None: """Print the named profiles the file defines, selected or not. - A named profile does nothing until a caller selects it, and that is the - thing readers of a configuration file get wrong: adding + A named profile does nothing until a caller selects it, and that is what + readers of a configuration file misunderstand: adding ``[waterdata.bulk]`` changes no run on its own. A report that mentioned a - profile only when one had been selected would leave that silence with - nothing to explain it -- the file would look ignored. + profile only when one had been selected would leave the profile's + lack of effect unexplained -- the file would appear to be ignored. Names come from the parsed file, so an unimported adapter's profiles are listed too. What such a profile *means* is what needs the import; what it - is called is a fact about the file, and withholding it here would make the - section's answer depend on which optional extras happened to be installed. + is called is a fact about the file, and omitting it here would make the + section's contents depend on which optional extras happened to be installed. """ defined = [ f"[{adapter}.{name}]" @@ -507,12 +509,12 @@ def _show_profiles(out: TextIO, parsed: _ParsedFile) -> None: def _show_unimported_adapters(out: TextIO) -> None: - """Name the adapters this process cannot report on, and say why. + """Name the adapters this process cannot report on, and state why. An adapter is only known to accept a setting once the module declaring that vocabulary has been imported, and NLDI is deliberately imported on demand for the geopandas extra. So the rows above cannot cover it. Omitting it - silently would read as "nothing is configured for nldi", which is a + would read as "nothing is configured for nldi", which is a different claim and an incorrect one -- this is the cost of validating an adapter's keys lazily (ADR 0011). """ @@ -549,10 +551,10 @@ def concurrency( """Cap on simultaneous chunks; ``None`` means unbounded. ``default`` is the caller's own preference for when nothing is configured -- - Water Use ships a lower figure than the OGC getters, because the NWDC is + Water Use uses a lower value than the OGC getters, because the NWDC is only stress-tested to that level. A value resolved from the chain always - wins over it: a service able to override an explicit setting would make - ``concurrency=1`` a lie. + takes precedence over it: a service able to override an explicit setting would make + ``concurrency=1`` untrue. """ raw, label, _source = _resolve("concurrency", adapter) if raw is None: @@ -586,8 +588,8 @@ def progress() -> bool | None: def parallel_chunks(*, adapter: str | None = None) -> int: """Configured default fan-out for multi-value queries. - ``1`` (the default) means "chunk only as much as the URL byte limit - forces". This is the *baseline*; + ``1`` (the default) chunks only as far as the URL byte limit forces. This is the + *baseline*; :func:`dataretrieval.parallel_chunks` overrides it for one call. Shares the name of that context manager because it is the same setting -- this is the resolved value, not the scoping block. @@ -622,21 +624,21 @@ def base_url(*, adapter: str | None = ..., default: str) -> str: ... def base_url(*, adapter: str | None = None, default: str | None = None) -> str | None: """An adapter's configured base URL, falling back to *default*. - Settable from code only: an adapter configuration may carry it, and both + Settable from code only: an adapter configuration may include it, and both the file and the environment refuse it -- the file at :func:`_accepted_keys` and the environment at :data:`_REFUSED_ENV_VARS`, each with an error naming - the block to write instead. A file that silently redirects a data-retrieval - library to another host is a supply-chain-shaped hazard, while a + the block to write instead. A file that redirects a data-retrieval + library to another host is a supply-chain hazard, while a ``configure`` block keeps the redirect where a reader of the script sees it (ADR 0011). - There is no package-wide default, because there is no one base URL: what an - adapter's requests are built on is the adapter's own fact, so the service - passes its own -- ``base_url(adapter="nldi", default=NLDI_API_BASE_URL)`` - -- and the URL stays declared beside the service that owns it. What lives - here is the *rule* for choosing between them, which was being spelled at - every read site as ``... or SERVICE_DEFAULT``; a change to it (normalizing - a trailing slash, say) is one edit rather than five. + There is no package-wide default, because there is no one base URL: the base an + adapter's requests use is the adapter's own setting, so the service passes its own + -- ``base_url(adapter="nldi", default=NLDI_API_BASE_URL)`` -- and the URL stays + declared in the module of the service that uses it. What is defined here is the + *rule* for choosing between them, which every read site restated as ``... or + SERVICE_DEFAULT``; a change to it (normalizing a trailing slash, say) is one edit + rather than five. Parameters ---------- @@ -644,8 +646,8 @@ def base_url(*, adapter: str | None = None, default: str | None = None) -> str | Whose base URL to resolve. default : str, optional The service's own base, returned when nothing configured one. Omitted, - the answer is ``None`` -- which is what :func:`show_configuration` asks - for, having no service default to name. + ``None`` is returned -- which is what :func:`show_configuration` passes, + since it has no service default to supply. """ raw, label, _source = _resolve("base_url", adapter) if raw is None: @@ -655,42 +657,41 @@ def base_url(*, adapter: str | None = None, default: str | None = None) -> str | # --- resolution ---------------------------------------------------------- -#: Which source of the chain answered a resolution. Machine-readable so a +#: Which source of the chain supplied a resolution. Machine-readable so a #: per-source rule reads the source, never the display label -- :func:`progress` -#: keys its legacy-lenient parsing on ``_ENV``, and the label stays purely -#: presentational. +#: keys its legacy-lenient parsing on ``_ENV``, and the label is only presentational. _BLOCK, _ENV, _FILE, _DEFAULT = "block", "environment", "file", "built-in" def _resolve(name: str, adapter: str | None = None) -> tuple[str | None, str, str]: """Return the raw value for *name*, its origin label, and its source. - Precedence is *source-major*: the chain walks block, then environment, then - file, exactly as ADR 0009 defines it -- and *within* each source an + Precedence is *source-major*: the chain checks block, then environment, then + file, as ADR 0009 defines it -- and *within* each source an adapter-scoped value outranks a package-wide one. So a variable exported - for one run still beats a stale ``[wqp]`` table in the config file -- + for one run still outranks a stale ``[wqp]`` table in the config file -- ordering by scope first, putting every adapter-scoped value ahead of every - package-wide one whatever its source, would have quietly inverted that + package-wide one whatever its source, would have inverted that (ADR 0010). ``adapter`` names the adapter on whose behalf the setting is being read. ``None`` resolves the package-wide value, which is also what an adapter - that declares no interest in this setting gets. + that does not read this setting gets. Returns ------- tuple[str or None, str, str] - The raw string as written (parsing happens per setting, so each keeps - its own blank-value rule), the human-readable origin label, and which - source answered (one of the constants above) -- ``None`` with - ``_BUILT_IN`` / ``_DEFAULT`` when nothing configured it. + The raw string as written (parsing happens per setting, so each keeps its own + blank-value rule), the human-readable origin label, and which source supplied + the value (one of the constants above) -- ``None`` with ``_BUILT_IN`` / + ``_DEFAULT`` when nothing configured it. """ _check_adapter_known(adapter) _check_env_not_refused(name) - # ``None`` unless this adapter actually reads this setting, so a setting - # outside its vocabulary resolves package-wide rather than looking for a - # scope it could never have been written into. + # ``None`` unless this adapter reads this setting, so a setting + # outside its vocabulary resolves package-wide rather than checking a + # scope it cannot have been written into. scoped: str | None = ( adapter if adapter is not None and _accepts(adapter, name) else None ) @@ -709,8 +710,8 @@ def _resolve(name: str, adapter: str | None = None) -> tuple[str | None, str, st def _check_adapter_known(adapter: str | None) -> None: """Raise if *adapter* is not in the configurable adapter roster. - An adapter name nobody recognizes is a typo in *our* source, and its - failure mode is silence: ``_accepts`` would wave every setting through, + An adapter name not in the roster is a typo in this package's source, and it + would fail without an error: ``_accepts`` would accept every setting, the file would hold no table under that name, and the read would fall through to the package-wide value -- so a ``[waterdata]`` table, or a ``WaterdataConfiguration``, would be ignored with nothing raised anywhere. @@ -725,10 +726,10 @@ def _check_adapter_known(adapter: str | None) -> None: def _check_env_not_refused(name: str) -> None: """Raise if an environment variable is set for a code-only setting. - Refused before anything is consulted, not at the environment's turn in - the chain. The file and the environment refuse ``base_url`` as one rule - (ADR 0011), so a variable that cannot work is not silently outranked by a - block that happens to work. + Refused before anything is consulted, not when the chain reaches the environment + source. The file and the environment refuse ``base_url`` as one rule (ADR 0011), so + a variable that cannot work is not outranked, with no error, by a block that happens + to work. """ refused = _REFUSED_ENV_VARS.get(name) if refused is not None and refused in os.environ: @@ -743,11 +744,11 @@ def _check_env_not_refused(name: str) -> None: def _resolve_from_block( name: str, scoped: str | None ) -> tuple[str | None, str, str] | None: - """Walk the scope stack for the first block that sets *name*. + """Check each scope frame, innermost first, for the first block that sets *name*. - Innermost block first: a value set by a nested block wins over both + Innermost block first: a value set by a nested block takes precedence over both scopes of an enclosing one. Within one block the adapter-scoped value is - the more specific of the two, so it is asked first. + the more specific of the two, so it is checked first. """ for frame in reversed(_scope.get()): if scoped is not None and (scoped, name) in frame: @@ -761,7 +762,7 @@ def _resolve_from_env(name: str) -> tuple[str | None, str, str] | None: """Check whether an environment variable supplies the setting. No per-adapter environment variables: seven adapters times four settings - is a namespace nobody can hold in mind, and an exported variable is + is a namespace nobody could remember, and an exported variable is invisible at the call site. See ADR 0010. """ env = ENV_VARS.get(name) @@ -794,7 +795,7 @@ def _resolve_from_file(name: str, scoped: str | None) -> tuple[str | None, str, def _accepts(adapter: str, name: str) -> bool: """Whether *adapter* reads the setting *name*. - An adapter this process has not imported has no vocabulary to consult, so + An adapter this process has not imported has declared no settings to check, so every setting is assumed to be in scope for it: the file stays valid either way, and an adapter cannot be misreading a setting it has not loaded. See :func:`settings_for`. @@ -820,13 +821,14 @@ def _display_progress(adapter: str | None = None) -> str: #: How each setting renders in :func:`show_configuration`. Keyed by the same #: names as :data:`_ALL_SETTINGS`, and asserted to cover them, so a setting -#: added to one without the other fails loudly instead of silently printing a -#: neighbour's value in the one report whose whole job is to be trustworthy. +#: added to one without the other fails at import instead of printing a +#: neighbouring setting's value in the one report that exists to report +#: provenance accurately. #: #: Every renderer takes the adapter to resolve for, so the adapter-override #: rows use this same table rather than a parallel one that the guard below #: would not cover. ``api_key`` and ``progress`` ignore it -- neither is -#: adapter-scoped, and :func:`_show_adapter_overrides` never asks them. +#: adapter-scoped, and :func:`_show_adapter_overrides` never calls them. _DISPLAYS: dict[str, Callable[[str | None], str]] = { "api_key": _display_api_key, "concurrency": _display_concurrency, @@ -839,7 +841,7 @@ def _display_progress(adapter: str | None = None) -> str: if set(_DISPLAYS) != set(_ALL_SETTINGS): # pragma: no cover - guards a coding error # Not an ``assert``: ``python -O`` strips those, and this guards the one - # report whose whole job is to be trustworthy about provenance. + # report that exists to report provenance accurately. raise RuntimeError( "every setting needs a show_configuration renderer; " f"missing={sorted(set(_ALL_SETTINGS) - set(_DISPLAYS))} " diff --git a/dataretrieval/credentials.py b/dataretrieval/credentials.py index 6e0b7ebdb..d30a6791d 100644 --- a/dataretrieval/credentials.py +++ b/dataretrieval/credentials.py @@ -1,12 +1,11 @@ -"""Which host honors the USGS API key, and how it is attached and withheld. +"""Which host accepts the USGS API key, and how it is attached and withheld. -One leaf owns every answer about the ``API_USGS_PAT`` credential: the host that -accepts it, whether a given destination qualifies, how it is stripped back off a -request bound somewhere else, and which keyword names are a caller *asking* to -send it. ADR 0006 assigns that sole ownership, and ADR 0010 keeps the key out -of every adapter's settings. +One leaf owns every rule about the ``API_USGS_PAT`` credential: the host that accepts +it, whether a given destination qualifies, how it is stripped back off a request bound +somewhere else, and which keyword names are a request to send it. ADR 0006 assigns that +sole ownership, and ADR 0010 keeps the key out of every adapter's settings. -This sits below HTTP mechanics and below progress reporting in the layers +This is below HTTP mechanics and below progress reporting in the layers contract. Its only first-party dependency is :mod:`dataretrieval.configuration` -- itself a standard-library-only leaf -- which supplies the key's *value*. @@ -21,40 +20,40 @@ from dataretrieval import configuration as _configuration #: Environment variable holding the USGS Water Data personal access token. -#: Taken from the chain that reads it rather than spelled again here -- the +#: Taken from the chain that reads it rather than restated here -- the #: same rule ``test_credential_policy_has_one_definition`` enforces for the -#: authorized host, and for the same reason: two copies stop agreeing silently. +#: authorized host, and for the same reason: two copies stop agreeing without any error. API_KEY_ENV = _configuration.ENV_VARS["api_key"] -#: Where to register for a key. Surfaced once, by the progress reporter, when a +#: Where to register for a key. Printed once, by the progress reporter, when a #: query against the authorized host runs without one -- unauthenticated callers -#: hit much lower rate limits (see the ``API_USGS_PAT`` note in the README). +#: are subject to much lower rate limits (see the ``API_USGS_PAT`` note in the README). SIGNUP_URL = "https://api.waterdata.usgs.gov/signup/" -#: The only host that honors the key. Every other service this package talks to +#: The only host that accepts the key. Every other service this package calls #: ignores it, so sending it there would leak a credential for no benefit. _AUTHORIZED_API_KEY_HOST = "api.waterdata.usgs.gov" #: Origin of the Water Data API, built from the authorized host rather than -#: spelled again. The host that serves these endpoints and the host that honors -#: the key are the same fact, and the failure mode of keeping two copies is -#: silent: the endpoint moves, the predicate does not follow, and either the key -#: quietly stops attaching or it is sent somewhere nobody authorized. The -#: adapters import this instead of restating the authority (enforced by +#: restated. The host that serves these endpoints and the host that accepts +#: the key are the same fact, and keeping two copies fails without an +#: error: the endpoint moves, the predicate is not updated, and either the key +#: stops attaching or it is sent somewhere nobody authorized. The +#: adapters import this instead of restating the definition (enforced by #: ``test_credential_policy_has_one_definition``). WATERDATA_BASE_URL = f"https://{_AUTHORIZED_API_KEY_HOST}" def accepts_api_key(target_url: str | httpx.URL | None) -> bool: - """Whether ``target_url`` names the host that honors :data:`API_KEY_ENV`. + """Whether ``target_url`` names the host that accepts :data:`API_KEY_ENV`. - The single answer to "does this destination get the key" -- used when + The one predicate for whether a destination gets the key -- used when attaching the credential, when stripping it back off at redirect time, and - when deciding whether "get an API key" is useful advice rather than noise, so + when deciding whether "get an API key" is advice this caller can act on, so the three can't drift apart. - The scheme has to be ``https``, not just the host (ADR 0009): a redirect or - a server-supplied next-page link can name ``http://`` on the very host that + The scheme has to be ``https``, not only the host (ADR 0009): a redirect or + a server-supplied next-page link can name ``http://`` on the host that is otherwise authorized. """ if target_url is None: @@ -67,23 +66,24 @@ def accepts_api_key(target_url: str | httpx.URL | None) -> bool: def without_embedded_credentials(url: httpx.URL) -> httpx.URL: - """Drop any ``user:pass@`` from a URL we were *handed* rather than built. + """Drop any ``user:pass@`` from a URL received from the server rather than built + here. A next-page link is data, not configuration, and ``httpx`` derives an ``Authorization: Basic`` header from userinfo in a URL (ADR 0009). No USGS - service authenticates that way, so stripping it costs a caller nothing. + service authenticates that way, so stripping it removes nothing a caller needs. """ return url.copy_with(userinfo=b"") if url.userinfo else url # Credential-shaped keyword names must never reach a getter's generic query # passthrough: URLs are retained by clients, proxies, logs, and response -# metadata. The predicate lives in this leaf rather than in any one adapter so +# metadata. The predicate is defined in this leaf rather than in any one adapter so # that ten getters cannot drift into ten spellings of it (ADR 0006). # # Matched as *substrings* of the separator-stripped name, not as exact names: # an exact-match list misses the spelling the library's own docs make most -# tempting -- ``x_api_key``, after the ``X-Api-Key`` header. +# likely -- ``x_api_key``, after the ``X-Api-Key`` header. _CREDENTIAL_MARKERS = ( "apikey", "authorization", @@ -97,7 +97,7 @@ def without_embedded_credentials(url: httpx.URL) -> httpx.URL: # Whole names that are credentials on their own but too short to match as # substrings without catching legitimate query parameters. # -# ``session`` is deliberately absent from both lists: it carries no secret, so +# ``session`` is deliberately absent from both lists: it holds no secret, so # rejecting it with a credentials message reports an incorrect reason, and as a # substring it claims part of a namespace the *server* owns -- any future query # parameter containing it would be unreachable behind that message. @@ -105,19 +105,17 @@ def without_embedded_credentials(url: httpx.URL) -> httpx.URL: def refuse_credential_keywords(names: Iterable[str]) -> None: - """Raise ``TypeError`` if any of *names* reads as a request for the key. - - For the ``**kwargs`` passthroughs -- Water Data's ``**queryables`` and - WQP's search filters -- where a name the caller invents is forwarded to the - server as a query parameter. Both call this rather than each keeping its - own list, so a spelling learned from one adapter's mistake is refused by - the other on the same day. - - A usability check, not a security control (ADR 0009). It answers the - caller who reasonably guesses that a credential goes here, with a - ``TypeError`` naming ``with configure(Configuration(api_key=...)):`` - instead of a token in a URL -- the bare call is a no-op, since - ``configure`` is a context manager. + """Raise ``TypeError`` if any of *names* is a credential-shaped name. + + For the ``**kwargs`` passthroughs -- Water Data's ``**queryables`` and WQP's search + filters -- where a name the caller invents is forwarded to the server as a query + parameter. Both call this rather than each keeping its own list, so a name added + because of one adapter's mistake is refused by the other as well. + + A usability check, not a security control (ADR 0009). It gives the caller who + guesses that a credential goes here a ``TypeError`` naming ``with + configure(Configuration(api_key=...)):`` instead of a token in a URL -- the bare + call is a no-op, since ``configure`` is a context manager. """ forbidden = set() for name in names: @@ -137,9 +135,9 @@ def refuse_credential_keywords(names: Iterable[str]) -> None: def api_key() -> str | None: """The configured token, or ``None``. - Lives here, next to the host check and + Defined here, next to the host check and :func:`strip_api_key_from_untrusted_host`, so reading the key and the rules - governing where it may travel stay in one module. The value itself resolves + governing where it may be sent stay in one module. The value itself resolves through :func:`dataretrieval.configuration.api_key`, so host scoping applies identically no matter which source supplied the key. """ diff --git a/dataretrieval/exceptions.py b/dataretrieval/exceptions.py index 96cc1d6f3..1ac4342c9 100644 --- a/dataretrieval/exceptions.py +++ b/dataretrieval/exceptions.py @@ -8,7 +8,7 @@ deterministic failure is :class:`NetworkError`; a recoverable failure that exhausts retries during fan-out is a resumable ``ServiceInterrupted``. -Most failures are an :class:`HTTPError` carrying the response ``.status_code``, +Most failures are an :class:`HTTPError` holding the response ``.status_code``, of which :class:`TransientError` (429 / 5xx) is the retryable subset. The rest aren't a plain status: :class:`RequestTooLarge` (with :class:`URLTooLong` / :class:`Unchunkable`), :class:`NetworkError` (a failed connection, per above), @@ -17,14 +17,14 @@ the one member that is not a request failure at all: it reports an unusable setting or config file, raised from wherever a setting is first resolved -- which, because resolution is lazy, is inside whichever getter runs first. The -*warning* side of the taxonomy lives here too: :class:`SkippedItemWarning` +*warning* side of the taxonomy is defined here too: :class:`SkippedItemWarning` (specialized by :class:`SkippedRatingWarning`) for a per-item skip inside a batched retrieval, and :class:`DataCurrencyWarning` for an upstream dataset that has stopped being updated. -This module has no third-party runtime dependencies -- ``httpx`` is imported only -for type checking. Any module can therefore import it without pulling in pandas -or httpx, and without risking an import cycle. +This module has no third-party runtime dependencies -- ``httpx`` is imported only for +type checking. Any module can therefore import it without importing pandas or httpx, and +without risking an import cycle. """ from __future__ import annotations @@ -60,13 +60,12 @@ class DataRetrievalError(Exception): """Base class for every ``dataretrieval`` error. - Almost every member is a failed request, and the read-anywhere fields below - describe one. The exception is :class:`ConfigurationError`, which reports a - configuration the library cannot use; it appears here because configuration - is resolved lazily on the request path, so it surfaces from inside a getter - and one ``except DataRetrievalError`` should cover it too. It carries no - status and is not retryable, so the branching idiom below routes it to the - final ``raise``. + Almost every member is a failed request, and the read-anywhere fields below describe + one. The exception is :class:`ConfigurationError`, which reports a configuration the + library cannot use; it appears here because configuration is resolved lazily on the + request path, so it is raised from inside a getter and one ``except + DataRetrievalError`` should cover it too. It has no status and is not retryable, so + the branching idiom below routes it to the final ``raise``. Catch it to handle any USGS or EPA service failure uniformly, and branch on the read-anywhere fields below without needing the concrete subclass:: @@ -90,8 +89,8 @@ class DataRetrievalError(Exception): #: HTTP status that triggered the error, or ``None`` for errors without one #: (connection failure, too-long URL, no data). Set by :class:`HTTPError`. status_code: int | None = None - #: Seconds the server asked us to wait before retrying (its ``Retry-After`` - #: header), or ``None`` when it gave no hint. Set by :class:`TransientError`. + #: Seconds the server specified to wait before retrying (its ``Retry-After`` + #: header), or ``None`` when it sent none. Set by :class:`TransientError`. retry_after: float | None = None #: Whether re-issuing the same request might succeed -- ``True`` for the #: transient HTTP statuses (429 / 5xx, :class:`TransientError`) and for @@ -100,11 +99,11 @@ class DataRetrievalError(Exception): # These errors get pickled back across process boundaries (a lithops / # multiprocessing worker returns whatever it raises). Default ``BaseException`` - # pickling rebuilds via ``cls(*args)``, which these subclasses can't survive -- - # keyword-only constructor fields, and ``ChunkInterrupted`` builds its message + # pickling rebuilds via ``cls(*args)``, which these subclasses cannot be rebuilt by + # -- keyword-only constructor fields, and ``ChunkInterrupted`` builds its message # internally. So reconstruct via ``__new__`` + the standard getstate/setstate - # protocol, bypassing ``__init__``; a subclass drops unpicklable state by - # overriding ``__getstate__`` (see ``ChunkInterrupted``). + # protocol, bypassing ``__init__``; a subclass drops unpicklable state by overriding + # ``__getstate__`` (see ``ChunkInterrupted``). def __reduce__(self) -> tuple[Any, ...]: return (_new_error, (self.__class__,), self.__getstate__()) @@ -135,7 +134,7 @@ class HTTPError(DataRetrievalError): ``except HTTPError as e: ... if e.status_code == 404``. :class:`TransientError` (429 / 5xx) is the retryable subset, and is itself an ``HTTPError``. The one exception to "a status is an ``HTTPError``" is a request the service rejects - as too long: it surfaces as :class:`URLTooLong` (a :class:`RequestTooLarge`), + as too long: it is raised as :class:`URLTooLong` (a :class:`RequestTooLarge`), *not* an ``HTTPError``. Catch :class:`DataRetrievalError` to be certain of spanning every failure. See :func:`error_for_status` for the full mapping. @@ -153,7 +152,7 @@ def __init__(self, message: str, *, status_code: int) -> None: class TransientError(HTTPError): - """A 429 or 5xx the server may serve on a later try. + """A 429 or 5xx the server may not repeat on a later attempt. :class:`RateLimited` covers 429 and :class:`ServiceUnavailable` covers 5xx. @@ -177,7 +176,7 @@ class TransientError(HTTPError): retryable: ClassVar[bool] = True - #: Canonical status a concrete transient stamps when built without an + #: Canonical status a concrete transient sets when built without an #: explicit ``status_code`` (:class:`RateLimited` = 429, #: :class:`ServiceUnavailable` = 503). ``TransientError`` itself is abstract #: and sets none, so constructing it bare requires ``status_code``. @@ -211,7 +210,7 @@ class ServiceUnavailable(TransientError): """A request was rejected with a server error (HTTP 5xx). Raised by both the legacy ``query`` path and the Water Data path, so a 5xx - surfaces as one type whichever subsystem issued the request. ``.status_code`` + is raised as one type whichever subsystem issued the request. ``.status_code`` holds the actual 5xx; it falls back to 503 only on a bare hand-construction. """ @@ -247,9 +246,8 @@ class Unchunkable(RequestTooLarge): Raised by the Water Data chunker when even the smallest reducible plan (every list axis at one atom per chunk, the filter at one clause per chunk) still exceeds the server's byte limit. Unlike - :class:`URLTooLong`, then, automatic splitting has already been tried and - exhausted. Shrink the input lists, simplify the filter, or split the call - manually. + :class:`URLTooLong`, automatic splitting has already been tried. Shrink the + input lists, simplify the filter, or split the call manually. """ @@ -263,8 +261,8 @@ class NetworkError(DataRetrievalError): response arrived to classify. Wraps the underlying ``httpx`` transport exception, preserved on - ``__cause__``. Worth retrying (:attr:`~DataRetrievalError.retryable` is - ``True``), but carries no ``.status_code`` because no response came back. + ``__cause__``. Retryable (:attr:`~DataRetrievalError.retryable` is + ``True``), but has no ``.status_code`` because no response came back. """ retryable: ClassVar[bool] = True @@ -311,14 +309,14 @@ def __str__(self) -> str: class DataCurrencyWarning(UserWarning): """An upstream dataset is frozen, retired, or no longer updated. - Distinct from ``DeprecationWarning``, which promises that a *name in this - package* is going away and gives the caller something to migrate to. Here + Distinct from ``DeprecationWarning``, which states that a *name in this + package* is being removed and gives the caller something to migrate to. Here the API is unchanged and there is nothing to migrate: the service's own data - has stopped moving, and only the caller can judge whether that matters. + has stopped being updated, and only the caller can decide whether that matters. It is a ``UserWarning`` for that reason. Emitting it as a ``DeprecationWarning`` meant a downstream project running - ``-W error::DeprecationWarning`` -- ordinary CI hygiene -- could not call + ``-W error::DeprecationWarning`` could not call the affected getters with their default arguments at all. """ @@ -336,8 +334,8 @@ class SkippedItemWarning(UserWarning): Transient failures (429 / 5xx / timeouts / connection drops) are never skipped -- they are retried and, if retries run out, raised as a resumable interruption. Rate limiting in particular is systematic, so - skipping there would silently drop most of a batch; that silent loss is - the failure mode this policy exists to prevent. + skipping there would drop most of a batch undetected; that undetected loss is + what this policy exists to prevent. A warning rather than a log line so it is visible by default. To make any skip fatal (strict all-or-nothing behavior):: @@ -354,7 +352,7 @@ class SkippedRatingWarning(SkippedItemWarning): :func:`dataretrieval.waterdata.get_ratings`. Emitted when a single STAC feature fails deterministically -- a stale - catalog entry (404 on its data asset), a feature carrying no data asset, + catalog entry (404 on its data asset), a feature with no data asset, a malformed RDB file. The failed feature's id is absent from the returned dict. See :class:`SkippedItemWarning` for the policy and how to escalate a skip to an error. @@ -396,28 +394,28 @@ def error_for_status( def parse_retry_after(value: str | None) -> float | None: - """Parse a ``Retry-After`` header into seconds, or ``None`` for no usable hint. + """Parse a ``Retry-After`` header into seconds, or ``None`` for no usable value. Both header forms mean the same thing and are treated the same way: the seconds are returned as given, however large. A value past what a caller will - wait out inline stops the retry and surfaces a transient carrying the hint on + wait out inline stops the retry and raises a transient with the value on ``.retry_after``, so a long wait becomes the caller's decision (and, for a chunked call, a resumable interruption) instead of being ignored. - An over-long hint is honored rather than discarded. Dropping it would make - the client retry *harder* against a service that just asked for a long - pause, and would deny the caller the number it needs on ``.retry_after``. - Clock skew can inflate a date-form hint, but trusting one costs a - recoverable escalation while ignoring it costs hammering a service that is - already asking for room. - - A date that has *already* passed yields no hint at all rather than ``0.0``. - Read literally it says "retry now", but the likelier reading is that our - clock runs ahead of the server's -- and acting on it would re-send almost - immediately against a service that just asked for a pause. Falling back to - our own bounded backoff is right under either reading. (Delta-seconds is - clock-independent, so a literal ``Retry-After: 0`` is still honored as the - instruction it is, floored by + An over-long value is kept rather than discarded. Dropping it would make + the client retry sooner against a service that just named a long delay, + and would deny the caller the number it needs on ``.retry_after``. Clock + skew can inflate a date-form value, but accepting one leads to a recoverable + escalation while ignoring it leads to re-sending to a service that has + already named a delay. + + A date that has *already* passed yields no value at all rather than + ``0.0``. Read literally it says "retry now", but the likelier reading is + that the client clock is ahead of the server's -- and acting on it would + re-send almost immediately against a service that just named a delay. + Falling back to the client's own bounded backoff is right under either + reading. (Delta-seconds is clock-independent, so a literal + ``Retry-After: 0`` is still applied as the instruction it is, floored by :meth:`~dataretrieval.transport.retry.RetryPolicy.backoff`'s jitter.) """ if not value: @@ -428,9 +426,9 @@ def parse_retry_after(value: str | None) -> float | None: except ValueError: pass else: - # ``inf``/``nan`` parse without error but poison every later comparison: an - # infinite hint would refuse retry forever and travel to the caller on - # ``.retry_after``. Treat them as no hint at all. + # ``inf``/``nan`` parse without error but make every later comparison + # meaningless: an infinite value would prevent retry forever and reach the + # caller on ``.retry_after``. Treat them as no value at all. return max(0.0, seconds) if math.isfinite(seconds) else None try: retry_at = parsedate_to_datetime(raw) diff --git a/dataretrieval/interruptions.py b/dataretrieval/interruptions.py index a7aa09a9b..a77e5ded0 100644 --- a/dataretrieval/interruptions.py +++ b/dataretrieval/interruptions.py @@ -1,14 +1,14 @@ """Resumable fan-out interruption exceptions — the public resume contract. -When a fanned-out request fails mid-stream (a 429, a 5xx, or a bare transport +When a fanned-out request fails partway (a 429, a 5xx, or a bare transport error), the work already completed is preserved and the call is resumable: the -raised exception carries a ``.call`` handle whose ``resume()`` re-issues only +raised exception has a ``.call`` handle whose ``resume()`` re-issues only the still-pending chunks. These exception types are that contract, re-exported at the top level (``from dataretrieval import ChunkInterrupted``). -The execution machinery that raises and resumes them is +The executor that raises and resumes them is :class:`dataretrieval.transport.fanout.FanOut`. -Vocabulary, consistently (see ``CONTEXT.md``): a **chunk** is one of the +Terms, as ``CONTEXT.md`` defines them: a **chunk** is one of the requests a query was split into, named for being a piece rather than for why it became one; **chunking** is how a query is split; and a **fan-out** is the concurrent execution of a query's chunks. Water Use chunks one request per @@ -18,15 +18,15 @@ ``ChunkInterrupted`` is retained as an alias of that same class, not a deprecated shim to delete later: it is the name published in the user guide and -caught in user code, and aliasing costs nothing to keep. ``except +caught in user code, and the alias has no maintenance cost. ``except ChunkInterrupted`` and ``except FanOutInterrupted`` are the same handler. This is a top-level leaf rather than a member of ``ogc`` or ``transport``, for the reason ADR 0006 gives for ``combining``, ``progress``, and ``credentials``: adapters need it whether or not they go through transport, and an exception taxonomy is not HTTP execution policy. It stays out of -:mod:`dataretrieval.exceptions` because it carries pandas/httpx state, which -would pull heavy dependencies into that lightweight leaf. +:mod:`dataretrieval.exceptions` because it holds pandas/httpx state, which +would make that leaf depend on pandas and httpx. """ from __future__ import annotations @@ -46,39 +46,37 @@ class FanOutInterrupted(DataRetrievalError): """ - Base class for mid-stream chunk failures whose completed work + Base class for partway chunk failures whose completed work is preserved and resumable. A ``FanOutInterrupted`` subclass means: a chunk failed, but - ``FanOut`` still owns whatever completed successfully before - the failure. Call ``self.call.resume()`` to pick up where the - failure stopped you — only still-pending chunks are + ``FanOut`` still holds whatever completed successfully before + the failure. Call ``self.call.resume()`` to continue from where the + failure stopped — only still-pending chunks are re-issued. - Subclasses describe *why* ``FanOut`` stopped so callers can - pick a retry policy: :class:`QuotaExhausted` for 429 (wait for the - rate-limit window), :class:`ServiceInterrupted` for 5xx (wait for - the upstream to recover). The ``.call`` handle is the same object - across every interruption of a single fanned-out call — frames - accumulate across retries. + Subclasses describe *why* ``FanOut`` stopped so callers can choose a retry policy: + :class:`QuotaExhausted` for 429 (wait for the rate-limit window), + :class:`ServiceInterrupted` for 5xx (wait for the upstream to recover). The + ``.call`` handle is the same object across every interruption of a single fanned-out + call — frames accumulate across retries. Attributes ---------- call : FanOut or None Resumable handle into the ``FanOut`` that raised this exception. ``None`` only on hand-constructed exceptions (test - fixtures), where ``.call``-derived accessors degrade to - empty/``None``. + fixtures), where ``.call``-derived accessors return empty/``None``. retry_after : float or None - Seconds the server suggested waiting (``Retry-After`` header). - ``None`` when the server gave no hint. + Seconds the server specified for waiting (``Retry-After`` header). + ``None`` when the server sent none. completed_chunks : int Number of chunks successfully completed before the failure. total_chunks : int Total chunks in the plan. partial_frame : pandas.DataFrame Combined frame of work completed by the moment this exception - was raised. Snapshot at raise time — does NOT advance on a + was raised. Snapshot at raise time — does not advance on a later ``call.resume()`` (use ``exc.call.partial_frame`` for the live view). partial_response : httpx.Response or None @@ -89,10 +87,9 @@ class FanOutInterrupted(DataRetrievalError): Examples -------- - Retry on any transient interruption, honoring the server's - ``Retry-After`` hint when present and falling back to a fixed wait - otherwise. Each new interruption keeps the already-completed work - intact — only the still-pending chunks are re-issued. + Retry on any transient interruption, applying the server's ``Retry-After`` value + when present and falling back to a fixed wait otherwise. Each new interruption keeps + the already-completed work intact — only the still-pending chunks are re-issued. .. code-block:: python @@ -137,14 +134,14 @@ def __init__( self.call = call self.retry_after = retry_after self.status_code = self._resolve_status_code(cause) - # Snapshot partial state at raise time so the exception stays a stable - # record of the failure moment: ``exc.partial_frame`` / - # ``.partial_response`` do NOT advance on a later ``call.resume()`` - # (that live view is on ``call.partial_frame`` / ``.partial_response``). - # This keeps each interruption in a resume loop a faithful record of - # what it saw, rather than every exception aliasing the shared call's - # advancing state. ``.copy()`` guards the single-chunk fast path, where - # the combined frame may be returned verbatim. + # Snapshot partial state at raise time so the exception stays a stable record of + # the failure moment: ``exc.partial_frame`` / ``.partial_response`` do not + # advance on a later ``call.resume()`` (that live view is on + # ``call.partial_frame`` / ``.partial_response``). This keeps each interruption + # in a resume loop an accurate record of the state when it was raised, rather + # than every exception aliasing the shared call's advancing state. ``.copy()`` + # protects the single-chunk fast path, where the combined frame may be returned + # verbatim. if call is None: self.partial_frame: pd.DataFrame = pd.DataFrame() self.partial_response: httpx.Response | None = None @@ -172,8 +169,8 @@ def _resolve_status_code(self, cause: BaseException | None) -> int | None: status: int | None = getattr(type(self), "_DEFAULT_STATUS", None) if status is not None or cause is None: return status - # The status is usually a few frames down: a typed error raised - # ``from`` the httpx failure that carried it. + # The status is usually further down the chain: a typed error raised + # ``from`` the httpx failure that held it. for current in _walk_causes(cause): found: int | None = getattr(current, "status_code", None) if found is not None: @@ -181,13 +178,13 @@ def _resolve_status_code(self, cause: BaseException | None) -> int | None: return None def __getstate__(self) -> dict[str, Any]: - # Drop the live FanOut before pickling: its ``.fetch`` is an + # Drop the FanOut before pickling: its ``.fetch`` is an # undecorated module function pickle can't reference by name, so the # interruption can't cross a process boundary with ``.call`` attached. - # The degraded ``call=None`` form keeps the counts, retry hint, and the + # The ``call=None`` form keeps the counts, retry-after value, and the # snapshotted partial frame / response — plain instance attributes the # base ``__getstate__`` already pickles; only ``.resume()`` is lost - # (cross-process resume was never possible anyway). + # (cross-process resume was never possible). return {**super().__getstate__(), "call": None} @@ -206,7 +203,7 @@ class QuotaExhausted(FanOutInterrupted): "HTTP 429 after {completed_chunks}/{total_chunks} chunks; " "catch QuotaExhausted (or FanOutInterrupted) to access " ".partial_frame or .call.resume() once the rate-limit " - "window has rolled over." + "window has reset." ) _DEFAULT_STATUS = 429 @@ -228,12 +225,12 @@ class ServiceInterrupted(FanOutInterrupted): ) -# Resolver failures that will not resolve differently on a later attempt. The -# temporary ones (notably EAI_AGAIN -- "try again", raised while a resolver is -# still coming up, on VPN reconnect, or after a laptop wakes) are deliberately -# absent: those are worth another try. Looked up defensively because the EAI_* -# constants are platform-dependent; an unrecognized code stays retryable, since -# spending a few seconds on a retry is cheaper than dropping a recoverable call. +# Resolver failures that will not resolve differently on a later attempt. The temporary +# ones (notably EAI_AGAIN -- "try again", raised while a resolver is still starting, on +# VPN reconnect, or after a laptop resumes from sleep) are deliberately absent: those +# are retried. Looked up with ``getattr`` defaults because the EAI_* constants are +# platform-dependent; an unrecognized code stays retryable, since a few seconds on a +# retry cost less than a dropped recoverable call. _PERMANENT_DNS_ERRORS = frozenset( code for code in ( @@ -256,8 +253,8 @@ def _walk_causes( ``__cause__`` (explicit ``raise ... from``) is always followed. ``__context__`` (implicit chaining, from raising inside an ``except`` block) is followed only when ``follow_context`` is set, because it can - lead away from the failure being classified into whatever unrelated error - happened to be in flight. + reach an unrelated error that happened to be in flight rather than the + failure being classified. The ``seen`` set keeps a chain that rejoins itself, or points back at an ancestor, from looping. @@ -282,21 +279,21 @@ def _deterministic_failure(exc: BaseException) -> bool: True for an unsupported scheme, a malformed request, or a hostname the resolver rejects permanently. A *temporary* resolver failure is not in that class and stays retryable (see :data:`_PERMANENT_DNS_ERRORS`). Bounding - retry to failures a later attempt could survive is ADR 0006. + retry to failures a later attempt might not repeat is ADR 0006. Walks ``__context__`` as well as ``__cause__``, because the original - failure is several layers down and not always an explicit ``raise ... - from``: a DNS failure reaches us as ``NetworkError`` -> + failure is several links down the chain and not always an explicit ``raise ... + from``: a DNS failure arrives as ``NetworkError`` -> ``httpx.ConnectError`` -> ``httpcore.ConnectError`` -> ``socket.gaierror``, - linked by implicit chaining. Following only the cause would walk off down - the explicit branch and miss a ``gaierror`` sitting on the implicit one -- + linked by implicit chaining. Following only the cause would follow + the explicit branch and miss a ``gaierror`` on the implicit one -- spending the whole retry budget on a hostname that will never resolve. """ for current in _walk_causes(exc, follow_context=True): if isinstance(current, (httpx.UnsupportedProtocol, httpx.LocalProtocolError)): return True if isinstance(current, socket.gaierror): - # Return, not continue: the first resolver code found settles the chain. + # Return, not continue: the first resolver code found decides. return current.errno in _PERMANENT_DNS_ERRORS return False @@ -312,7 +309,7 @@ def _classify_transient( if isinstance(exc, (httpx.HTTPError, httpx.InvalidURL)): # Some failures will fail the same way every time -- an unsupported # scheme, a hostname that doesn't resolve. Offering to resume one - # would hide the real error behind a retry that can never work. + # would replace the underlying error with a retry that can never succeed. if _deterministic_failure(exc): return None return ServiceInterrupted, None diff --git a/dataretrieval/ngwmn.py b/dataretrieval/ngwmn.py index d44cf8dfd..ced8cc79f 100644 --- a/dataretrieval/ngwmn.py +++ b/dataretrieval/ngwmn.py @@ -6,7 +6,7 @@ ``providers``. Each getter below delegates to the shared OGC facade (:func:`~dataretrieval.ogc.get_ogc_data`) with ``base_url=NGWMN_OGC_API_URL``. Multi-value chunking, pagination, retry/resume, and result shaping therefore -behave exactly as they do for the main Water Data getters. +behave as they do for the main Water Data getters. Unlike the main Water Data collections, NGWMN aggregates monitoring locations from many agencies, so ``monitoring_location_id`` values use other agency @@ -58,17 +58,17 @@ NGWMN_OGC_API_URL = f"{BASE_URL}/ngwmn/ogcapi" # --- state-filter shim ------------------------------------------------------- -# NGWMN's collections expose DIFFERENT state queryables: ``sites`` filters on +# NGWMN's collections expose different state queryables: ``sites`` filters on # the full ``state_name`` (e.g. "Wisconsin"), while ``providers`` filters on the # two-letter postal ``state`` (uppercase, e.g. "WI"). The state-aware getters # take a single ``state`` parameter accepting any US-state encoding (full name, # postal code, or FIPS code); ``_get`` resolves it into the one queryable each -# collection wants via the shared ``codes.states.apply_state``, keyed by +# collection accepts via the shared ``codes.states.apply_state``, keyed by # ``_STATE_QUERYABLE`` below. # -# This shim exists only to smooth over that upstream asymmetry. +# This shim exists only to handle that upstream asymmetry. # ``tests/ngwmn_test.py::test_state_queryables_still_diverge_upstream`` fails -- -# the signal to remove it -- if the API ever unifies the two queryables. +# the indication that it can be removed -- if the API ever unifies the two queryables. _STATE_QUERYABLE = { # service -> ``apply_state`` kwargs (destination queryable + to_state format) "sites": {"into": "state_name", "to": "name"}, @@ -101,7 +101,7 @@ def _get(service: str, local_vars: dict[str, Any]) -> tuple[pd.DataFrame, BaseMe """Marshal a getter's arguments and dispatch to the shared OGC facade. Every NGWMN getter ends with this same call; centralizing it keeps the - NGWMN base URL, output id, and dialect wired up in exactly one place. + NGWMN base URL, output id, and dialect set in one place. """ queryable = _STATE_QUERYABLE.get(service) if queryable is not None: @@ -451,8 +451,8 @@ class NgwmnConfiguration( NGWMN is a second OGC API on the Water Data host, so its queries divide along the same URL byte budget and take the same two fan-out - dials. The API key is not among them: one gateway fronts both - adapters, so one key and one quota pool serve them (ADR 0010). + settings. The API key is not among them: one gateway fronts both + adapters, so they share one key and one quota pool (ADR 0010). Declared here rather than in :mod:`dataretrieval.configuration` (ADR 0011). @@ -468,7 +468,7 @@ class NgwmnConfiguration( OGC API base to send NGWMN requests to, instead of the service's own (``NGWMN_OGC_API_URL``). Code only: the file and the environment refuse it. The API key is scoped to the host that - honors it, so a redirected call carries no key. + accepts it, so a redirected call sends no key. concurrency : int or str, optional Cap on simultaneous sub-requests, or ``"unbounded"``. parallel_chunks : int, optional diff --git a/dataretrieval/nldi.py b/dataretrieval/nldi.py index 3e12faa07..33b92027a 100644 --- a/dataretrieval/nldi.py +++ b/dataretrieval/nldi.py @@ -57,7 +57,7 @@ #: Built from the tuple above, so a mode added there cannot go unmentioned. _NAVIGATION_MODES_HINT = f"Pass one of {render_options(_VALID_NAVIGATION_MODES)}." #: Shared by the conflict check and the nothing-supplied check, so both -#: offer the same ways forward. +#: offer the same remedy. _ORIGIN_HINT = ( "Navigate from a comid, e.g. comid=13294314, or from a " "feature_source/feature_id pair -- not both" @@ -71,7 +71,7 @@ def _api_base() -> str: Every URL below is built from this rather than from :data:`NLDI_API_BASE_URL` directly, so a ``NldiConfiguration(base_url=...)`` - reaches every navigation, basin, and catalog request alike (ADR 0011). + applies to every navigation, basin, and catalog request alike (ADR 0011). """ return _configuration.base_url(adapter="nldi", default=NLDI_API_BASE_URL) @@ -80,9 +80,9 @@ def _query_nldi( url: str, query_params: dict[str, str], ) -> dict[str, Any] | list[Any]: - # A helper function to query the NLDI API. ``query()`` already raises a - # typed ``DataRetrievalError`` for any HTTP error response, so a returned - # response is a success that we only need to parse. + # A helper function to query the NLDI API. ``query()`` already raises a typed + # ``DataRetrievalError`` for any HTTP error response, so a returned response is a + # success that only needs parsing. response = _query_with_retry(url, payload=query_params, adapter="nldi") response_data: dict[str, Any] | list[Any] = {} try: @@ -95,13 +95,14 @@ def _query_nldi( def _features_to_gdf(feature_collection: dict[str, Any]) -> gpd.GeoDataFrame: - """Build a GeoDataFrame from an NLDI FeatureCollection, tolerating empties. - - NLDI can legitimately return no features (e.g. a feature with nothing - upstream), and :func:`_query_nldi` returns ``{}`` when a 200 response - carries no JSON body. ``GeoDataFrame.from_features`` raises on both cases - (there's no geometry column to attach the CRS to), so return an empty - GeoDataFrame carrying ``_CRS`` instead of crashing. + """Build a GeoDataFrame from an NLDI FeatureCollection, accepting an empty + collection. + + NLDI can validly return no features (e.g. a feature with nothing upstream), and + :func:`_query_nldi` returns ``{}`` when a 200 response has no JSON body. + ``GeoDataFrame.from_features`` raises on both cases (there's no geometry column to + attach the CRS to), so return an empty GeoDataFrame with ``_CRS`` set instead of + raising. """ features = feature_collection.get("features") if feature_collection else None if not features: @@ -336,10 +337,10 @@ def _navigation_request( ) -> tuple[str, dict[str, str]]: """URL and query params for an NLDI navigation from a validated origin. - The single home for the navigation path grammar — ``{origin}/navigation/ - {mode}/{tail}`` — and its ``distance`` knob. Callers add the knobs specific - to their endpoint (``trimStart``, ``stopComid``) afterwards, so the query - string keeps its documented parameter order. + The one definition of the navigation path grammar — ``{origin}/navigation/ + {mode}/{tail}`` — and its ``distance`` parameter. Callers add the + parameters specific to their endpoint (``trimStart``, ``stopComid``) + afterwards, so the query string keeps its documented parameter order. """ origin = f"{feature_source}/{feature_id}" if feature_source else f"comid/{comid}" url = f"{_api_base()}/{origin}/navigation/{navigation_mode}/{tail}" @@ -400,10 +401,10 @@ def _get_features_request( return f"{_api_base()}/{feature_source}/{feature_id}", {} # Before the data_source check below: a caller who mistyped the mode should - # hear about the mode, not be sent to fix a second argument first. + # be told about the mode, not told to fix a second argument first. navigation_mode = _validate_navigation_mode(navigation_mode) - # The navigation's tail is the data source, so a missing one is spelled - # "None" into the path and the service answers 200 with zero features. + # The navigation's tail is the data source, so a missing one is written as + # "None" into the path and the service returns 200 with zero features. data_source = require_argument( "data_source", data_source, @@ -451,7 +452,7 @@ def get_features_by_data_source(data_source: str) -> gpd.GeoDataFrame: .. doctest:: >>> # "nwissite" returns every NWIS site nationwide, so this example is - >>> # skipped in the doctest build to avoid the (very large) download. + >>> # skipped in the doctest build to avoid the large download. >>> gdf = dataretrieval.nldi.get_features_by_data_source( # doctest: +SKIP ... data_source="nwissite" ... ) @@ -656,7 +657,7 @@ def _validate_data_source(data_source: str, *, name: str = "data source") -> Non "NLDI data-source catalog returned an unexpected shape; " "expected a list of {'source': ..., ...} objects, got: " f"{available_data_sources!r}. If you set " - "NldiConfiguration(base_url=...), point it at the linked-data " + "NldiConfiguration(base_url=...), set it to the linked-data " "root, e.g. base_url='https://api.water.usgs.gov/nldi/" "linked-data'; otherwise the service returned an unexpected " "body -- retry later." @@ -681,8 +682,9 @@ def _validate_feature_source_comid( feature_source: str | None, feature_id: str | None, comid: int | None ) -> None: if comid is not None: - # Half a feature pair beside a comid is a conflict, not a gap: advising - # the caller to complete the pair would only raise the conflict next. + # Half a feature pair with a comid is a conflict, not a missing argument: + # advising the caller to complete the pair would only raise the conflict + # next. reject_together( { "comid": comid, @@ -708,11 +710,11 @@ def _validate_feature_source_comid( class NldiConfiguration(_Redirectable, _Retrying, BaseConfiguration): """Settings for NLDI calls alone. - No fan-out dials: an NLDI query is answered by a single request. + No fan-out settings: an NLDI query is served by a single request. This adapter is imported on demand for the geopandas extra, so this - class registers itself later than the rest -- which is exactly why - the adapter roster lives in :data:`~dataretrieval.configuration.ADAPTERS` + class registers itself later than the rest -- which is why + the adapter roster is :data:`~dataretrieval.configuration.ADAPTERS` rather than being derived from what has been imported. Declared here rather than in :mod:`dataretrieval.configuration` diff --git a/dataretrieval/nwdc.py b/dataretrieval/nwdc.py index dd38b5283..05efcc7b7 100644 --- a/dataretrieval/nwdc.py +++ b/dataretrieval/nwdc.py @@ -10,7 +10,7 @@ Unlike the main Water Data getters (:mod:`dataretrieval.waterdata`) and NGWMN (:mod:`dataretrieval.ngwmn`), the NWDC is a plain CSV REST service rather than -an OGC API Features collection. This module supplies the NWDC-specific bits -- +an OGC API Features collection. This module supplies the NWDC-specific parts -- request building, CSV parsing, the ``Link``-header cursor, and the ``{detail}`` error envelope -- over the service-neutral transport layer (ADR 0006). @@ -92,15 +92,15 @@ #: This service's preferred in-flight cap when nothing is configured. Lower #: than the package default of 32 because every location retries -#: independently, so a rate-limit episode bursts this number times the retry -#: count; the NWDC tolerates this level without rate-limit errors (verified by +#: independently, so a rate-limit episode produces this number times the retry +#: count of requests; the NWDC returns no rate-limit errors at this level (verified by #: stress test) and higher has not been tested. Any configured concurrency #: overrides it -- see :func:`dataretrieval.configuration.concurrency` for why the #: general setting outranks a module's default rather than the reverse. DEFAULT_CONCURRENT_REQUESTS = 4 -# Page responses carry the HUC12 identifier in this column; it must stay a -# string so leading zeros (e.g. "010900020502") survive the round trip. +# Page responses hold the HUC12 identifier in this column; it must stay a +# string so leading zeros (e.g. "010900020502") are preserved through parsing. _HUC12_COLUMN = "huc12_id" @@ -124,7 +124,7 @@ def get_wateruse( ``state``, ``county``, or ``huc``; results are always returned on a HUC12 grid, in a long (tidy) frame with one row per HUC12 and time step. Large areas (e.g. a whole region or a populous state) are served across multiple - pages; this function follows those pages transparently and concatenates + pages; this function follows those pages automatically and concatenates them into one frame. Each selector also accepts a list of values. The NWDC queries one area per @@ -152,12 +152,11 @@ def get_wateruse( groundwater and surface-water components). Multiple variables are comma-joined into a single request. The service requires at least one variable; omitting it returns a 400 listing the model's valid variable - IDs (surfaced as a :class:`~dataretrieval.exceptions.DataRetrievalError`). + IDs (raised as a :class:`~dataretrieval.exceptions.DataRetrievalError`). state : string, int, or iterable, optional One or more US states/territories to query. Each accepts a full name - (``"Wisconsin"``), a two-letter postal code (``"WI"``), or a two-digit - ANSI/FIPS code (``"55"`` or ``55``), mirroring - :func:`dataretrieval.ngwmn.get_sites`. + (``"Wisconsin"``), a two-letter postal code (``"WI"``), or a two-digit ANSI/FIPS + code (``"55"`` or ``55``), matching :func:`dataretrieval.ngwmn.get_sites`. county : string or iterable, optional One or more five-digit county FIPS codes — state FIPS + county FIPS, e.g. ``"55025"`` for Dane County, Wisconsin. @@ -247,7 +246,7 @@ def get_wateruse( base_params = {k: v for k, v in base_params.items() if v is not None} # An ``NwdcConfiguration(base_url=...)`` from an enclosing block, or this - # service's own endpoint, threaded through every request and the page walk + # service's own endpoint, passed to every request and the page walk # (ADR 0011). service_url = _configuration.base_url(adapter="nwdc", default=WATERUSE_URL) @@ -311,7 +310,7 @@ def _as_list(value: object) -> list[Any]: A scalar becomes a one-element list; any non-string iterable (list, tuple, Series, ndarray, generator) is materialized to a list. A string is treated - as a scalar so it isn't exploded into characters. + as a scalar so it is not split into characters. """ if isinstance(value, Iterable) and not isinstance(value, str): return list(value) @@ -351,15 +350,15 @@ def _fan_out( This function is only the NWDC-specific half: parse a CSV page and read its ``Link`` header cursor, follow that cursor, raise the typed error - carrying the NWDC ``detail``, and shape the result. + that includes the NWDC ``detail``, and shape the result. :func:`~dataretrieval.transport.pagination.run_paginated` owns the rest. The plan is the request list itself: the NWDC accepts one ``location=`` - per request, so the caller's locations arrive already separate (ADR 0008). + per request, so the caller's locations are already separate (ADR 0008). - The broad retry status set is on purpose: NWDC reports an invalid query as a 400 + The broad retry status set is deliberate: NWDC reports an invalid query as a 400 with a ``{"detail": ...}`` envelope, so unlike WQP and StreamStats its 5xx - really is an upstream fault worth re-sending. + is an upstream fault that re-sending can resolve. """ def parse(response: httpx.Response) -> tuple[pd.DataFrame, str | None]: @@ -411,18 +410,17 @@ def _next_page_url( """Return the absolute URL of the next page, or None if this is the last. Reads the standard ``Link: <...>; rel="next"`` header (parsed by httpx into - ``response.links``). The cursor is normalized before it is trusted, because - the service spells it inconsistently. A relative reference is resolved - against the page it came from, and the bare ``water.usgs.gov`` host is - rewritten to the public ``api.water.usgs.gov`` gateway (over https, whatever - scheme the link used) so the follow-up request reaches the API. Only a - cursor that still points somewhere else after that is refused -- following - it would send Water Use requests, and any credentials on them, to a host the - caller never asked for. - - ``host`` is the host this call is actually talking to, which is not the + ``response.links``). The cursor is normalized before it is used, because the service + writes it inconsistently. A relative reference is resolved against the page it came + from, and the bare ``water.usgs.gov`` host is rewritten to the public + ``api.water.usgs.gov`` gateway (over https, whatever scheme the link used) so the + follow-up request is sent to the API. Only a cursor that still names another host + after that is refused -- following it would send Water Use requests, and any + credentials on them, to a host the caller never asked for. + + ``host`` is the host this call sends requests to, which is not the NWDC's when a ``configure`` block redirected the adapter. The alias list and - the rewrite are facts about *this* service -- nothing else answers for + the rewrite are facts about *this* service -- no other host serves ``water.usgs.gov`` -- so a redirected call gets the general rule instead: follow a link only back to the host that served the page. Applying the NWDC's rewrite there would send page two of a mirrored query to the USGS. @@ -446,7 +444,7 @@ def _nwdc_error_detail(response: httpx.Response) -> str | None: The NWDC reports errors as ``{"detail": "Invalid model name: ..."}``. Passed to :func:`~dataretrieval.utils._raise_for_status` as ``detail_from`` so the - service's wording surfaces in the typed error message. + service's wording appears in the typed error message. """ try: body = response.json() @@ -454,7 +452,7 @@ def _nwdc_error_detail(response: httpx.Response) -> str | None: return None detail = body.get("detail") if isinstance(body, dict) else None if not isinstance(detail, str): - # A validation envelope spells ``detail`` as a list of error objects; + # A validation envelope gives ``detail`` as a list of error objects; # only prose belongs in a message. return None if detail.startswith("Invalid model name"): diff --git a/dataretrieval/nwis.py b/dataretrieval/nwis.py index 701f7a228..fec805f0f 100644 --- a/dataretrieval/nwis.py +++ b/dataretrieval/nwis.py @@ -43,13 +43,13 @@ WATERSERVICES_SERVICES = ["dv", "iv", "site", "stat"] # What ``get_record`` routes, which is wider than what ``query_waterdata`` -# reaches: 'ratings' is served by ``get_ratings`` from a different endpoint. +# serves: 'ratings' is served by ``get_ratings`` from a different endpoint. WATERDATA_SERVICES = [ "peaks", "ratings", ] -# The major filters each query function accepts, hoisted beside the service -# lists so the checks and their remedies read from one roster. +# The major filters each query function accepts, placed with the service +# lists so the checks and their remedies use one list. _NWIS_WEB_MAJOR_FILTERS = ("site_no", "stateCd") _NWIS_WEB_BBOX_CORNERS = ( "nw_longitude_va", @@ -88,7 +88,7 @@ def _warn_deprecated(func_name: str) -> None: - """Emit a per-function DeprecationWarning pointing at the waterdata replacement.""" + """Emit a per-function DeprecationWarning naming the waterdata replacement.""" warn_deprecated( f"`nwis.{func_name}`", replacement=_REPLACEMENTS[func_name], @@ -220,14 +220,14 @@ def preformat_peaks_response(df: pd.DataFrame) -> pd.DataFrame: Notes ----- An empty frame with no ``peak_dt`` column is returned unchanged, so an - empty peaks response reaches :func:`format_response`'s empty-frame path + empty peaks response takes :func:`format_response`'s empty-frame path instead of raising ``KeyError``. NWIS zero-fills the unknown part of a historical peak's date -- ``YYYY-MM-00`` when the day is not known, ``YYYY-00-00`` when the month is not either. Neither parses, so ``datetime`` is ``NaT`` for those peaks rather than a date NWIS does not have. The peak is kept regardless, and - ``peak_dt`` is left in the frame: the response carries no ``water_yr``, so + ``peak_dt`` is left in the frame: the response has no ``water_yr``, so it is the only column holding a censored peak's year. """ @@ -527,9 +527,9 @@ def _get_json_values( """Shared body of the JSON waterservices time-series getters (dv / iv). The caller-facing ``sites`` / ``start`` / ``end`` arguments are aliases: an - explicit waterservices keyword of the same meaning wins over them. Note that - ``multi_index`` travels through ``kwargs`` so that :func:`format_response` - sees it. + explicit waterservices keyword of the same meaning takes precedence over + them. Note that ``multi_index`` is passed through ``kwargs`` so that + :func:`format_response` sees it. """ _check_sites_value_types(sites) @@ -1064,7 +1064,7 @@ def get_record( def _site_block_boundaries(site_list: list[str]) -> list[int]: - """Return indices where the site number changes, bookended by 0 and len. + """Return indices where the site number changes, with 0 and len added at the ends. For example, given ``['A', 'A', 'B']`` returns ``[0, 2, 3]``. """ diff --git a/dataretrieval/ogc/__init__.py b/dataretrieval/ogc/__init__.py index 025d3fb12..1f43cb920 100644 --- a/dataretrieval/ogc/__init__.py +++ b/dataretrieval/ogc/__init__.py @@ -8,9 +8,9 @@ including verbatim-CQL2 queries via its ``cql_body`` parameter. Collection adapters (NGWMN, Water Data's generic wrapper) import from this -facade rather than reaching into engine internals — every name here is usable -through the facade alone. Generic execution policy lives in -:mod:`dataretrieval.transport`, which the engine now calls directly. +facade rather than importing engine internals — every name here is usable +through the facade alone. Generic execution policy is in +:mod:`dataretrieval.transport`, which the engine calls directly. """ from dataretrieval.ogc.engine import get_ogc_data diff --git a/dataretrieval/ogc/chunking.py b/dataretrieval/ogc/chunking.py index 00b1e163b..1bb867aaa 100644 --- a/dataretrieval/ogc/chunking.py +++ b/dataretrieval/ogc/chunking.py @@ -1,17 +1,16 @@ """URL-byte chunk planning and dispatch for the OGC getters. -An OGC query has several chunkable axes: every multi-value list -parameter (sites, parameter codes, …) plus the cql-text ``filter``, -which splits along its top-level OR clauses. Any of them can fan the -URL past the server's ~8 KB byte limit. ``ChunkPlan`` picks a fan-out -for each axis that minimizes total chunks while keeping every -chunk URL under the budget. Requests that already fit get a single-chunk -plan — the executor has one code path either way. +An OGC query has several chunkable axes: every multi-value list parameter (sites, +parameter codes, …) plus the cql-text ``filter``, which splits along its top-level OR +clauses. Any of them can make the URL exceed the server's ~8 KB byte limit. +``ChunkPlan`` picks a fan-out for each axis that minimizes total chunks while keeping +every chunk URL under the budget. Requests that already fit get a single-chunk plan — +the executor has one code path either way. This module owns the OGC-specific half: the byte budget, the -``parallel_chunks`` dial, and the ``multi_value_chunked`` decorator that -ties a plan to a fetcher. It hands the plan to -:class:`dataretrieval.transport.fanout.FanOut`, which drives the chunks to +``parallel_chunks`` setting, and the ``multi_value_chunked`` decorator that +ties a plan to a fetcher. It passes the plan to +:class:`dataretrieval.transport.fanout.FanOut`, which runs the chunks to completion; :class:`~dataretrieval.ogc.planning.ChunkPlan` satisfies :class:`~dataretrieval.transport.fanout.FanOutPlan` structurally. That split is ADR 0008. @@ -19,8 +18,8 @@ Parallel chunks: the planner is conservative by default — it splits only as far as the byte limit forces. A caller who knows their result is large can opt into a finer split via the ``parallel_chunks(n)`` context manager, which fans -the query out into ``n`` parallel chunks. ``n`` drives -:meth:`ChunkPlan._refine`; see ``parallel_chunks`` for the why and the when. +the query out into ``n`` parallel chunks. ``n`` is the input to +:meth:`ChunkPlan._refine`; see ``parallel_chunks`` for when to use it. Concurrency, retries, and interruption semantics are documented on :mod:`dataretrieval.transport.fanout`; the ``concurrency`` and ``retries`` @@ -57,14 +56,14 @@ from .planning import ChunkPlan -# Compatibility aliases for the chunking/progress test modules. The client -# names bind the *same* objects transport publishes, not copies -- a test -# reading a copy here would never see the running client. +# Compatibility aliases for the chunking/progress test modules. The client names bind +# the same objects transport sets, not copies -- a test reading a copy here would never +# see the running client. ChunkedCall = FanOut get_active_client = active_client _chunked_client = _active_client -# Empirically the API replies HTTP 414 above ~8200 bytes of full URL — +# Empirically the API returns HTTP 414 above ~8200 bytes of full URL — # matches nginx's default ``large_client_header_buffers`` of 8 KB. 8000 # leaves ~200 bytes for request-line framing and proxy variance. The decorator # resolves this module-level default at call time when ``url_limit`` is None, @@ -77,21 +76,19 @@ def parallel_chunks(n: int) -> Iterator[None]: """ Fan the OGC getters' multi-value requests out into ``n`` parallel chunks. - By default the Water Data / NGWMN getters chunk a request only as much as - the server's ~8 KB URL-byte limit forces — the fewest chunks that - fit. That default can be more conservative than a large pull needs. - Because every chunk paginates, splitting a large result further costs - little or no extra quota *as long as each chunk still spans many - pages* — rows-per-chunk far exceeding the page size (ten states pulled as - one request page nearly as many times as ten per-state requests would). - When a split leaves each chunk only a page or two, its partial final - page is extra, so finer chunks do add some requests. This context manager - lets a caller who *knows* their pull is large ask for that finer split. The - trade is roughly the same pages for more, smaller chunks, which gives - smoother progress, more even concurrency, and a smaller unit of - retry/resume. - - A per-call knob rather than an environment variable, and scoped to a + By default the Water Data / NGWMN getters chunk a request only as much as the + server's ~8 KB URL-byte limit forces — the fewest chunks that fit. That default can + split less than a large pull benefits from. Because every chunk paginates, splitting + a large result further costs little or no extra quota *as long as each chunk still + spans many pages* — rows-per-chunk far exceeding the page size (ten states pulled as + one request page nearly as many times as ten per-state requests would). When a split + leaves each chunk only a page or two, its partial final page is extra, so finer + chunks do add some requests. This context manager lets a caller who knows their pull + is large request that finer split. The result is roughly the same pages in more, + smaller chunks, which gives smoother progress, more even concurrency, and a smaller + unit of retry/resume. + + A per-call setting rather than an environment variable, and scoped to a ``with`` block: ADR 0009. Outside any block the getters use the conservative default. Only the OGC getters (Water Data, NGWMN) read this; wrapping a legacy NWIS call in the block is a no-op. @@ -99,17 +96,15 @@ def parallel_chunks(n: int) -> Iterator[None]: Parameters ---------- n : int - The number of chunks to fan the whole call out into — a positive - integer such as ``2``, ``8``, or ``32``. It caps the plan's *total* - chunk count (the cartesian product across every multi-value - argument combined, not per argument), so several multi-value arguments - cannot multiply past it. The cap is a ceiling, never exceeded: the - actual count is bounded below by what the ~8 KB URL limit already - forces and above by ``n``. So an ``n`` larger than the input allows - yields one chunk per value, and with several multi-value - arguments the total may land somewhat below ``n`` because splits are - whole (the plan can't always divide evenly onto ``n``). ``n=1`` asks - for no extra fan-out. + The number of chunks to fan the whole call out into — a positive integer such as + ``2``, ``8``, or ``32``. It caps the plan's *total* chunk count (the cartesian + product across every multi-value argument combined, not per argument), so + several multi-value arguments cannot multiply past it. The cap is a ceiling, + never exceeded: the actual count is bounded below by what the ~8 KB URL limit + already forces and above by ``n``. So an ``n`` larger than the input allows + yields one chunk per value, and with several multi-value arguments the total may + be somewhat below ``n`` because splits are whole (the plan can't always divide + evenly onto ``n``). ``n=1`` requests no extra fan-out. Each chunk fetches at least one page, so it costs at least one request against your hourly rate limit — a larger ``n`` spends more @@ -126,27 +121,27 @@ def parallel_chunks(n: int) -> Iterator[None]: ------ ValueError If ``n`` is not a positive integer — raised on ``with`` entry, before - any request is issued, so an invalid value fails loudly rather than silently + any request is issued, so an invalid value raises rather than doing nothing. Notes ----- - Fanning out carries the same consequences as the byte-limit chunking the - getters already do for oversized requests; opting in just brings them to a + Fanning out has the same consequences as the byte-limit chunking the + getters already do for oversized requests; opting in applies them to a request that would otherwise be a single call: - ``max_rows``: each chunk paginates up to ``max_rows`` rows independently, then the combined result is sorted and truncated to ``max_rows``. So a call with ``max_rows`` set returns a *different* (though still valid and deterministically sorted) row set inside a - ``parallel_chunks`` block than without one. The cap is drawn from the - union of the chunks, not a single stream. Don't pair a tight + ``parallel_chunks`` block than without one. The cap applies to the + union of the chunks, not to one sequence. Do not combine a small ``max_rows`` preview with ``parallel_chunks`` if you need exactly the rows the un-fanned call would return. - Resumability: a single request either fully succeeds or fully fails, but a fanned-out call can fail partway (e.g. a mid-call rate-limit) and raise a resumable :class:`~dataretrieval.interruptions.ChunkInterrupted` - (or ``QuotaExhausted``) carrying the completed chunks. Finish the + (or ``QuotaExhausted``) holding the completed chunks. Finish the call with ``exc.call.resume()``. - Cross-chunk de-duplication keys on the feature ``id``; features with no ``id`` can't be deduped, so overlapping filter clauses split @@ -182,16 +177,16 @@ def multi_value_chunked( adapter: str | None = None, ) -> Callable[[_Fetch[dict[str, Any]]], Callable[..., tuple[pd.DataFrame, Any]]]: """ - Decorate an async fetcher to transparently chunk over-budget requests. + Decorate an async fetcher to chunk over-budget requests automatically. Returns a callable that builds a :class:`ChunkPlan` from ``args``, constructs a :class:`ChunkedCall` over the decorated - ``async def fetch(args) -> (df, response)``, and drives it to + ``async def fetch(args) -> (df, response)``, and runs it to completion via :meth:`ChunkedCall.resume`. The plan splits multi-value list params and the cql-text filter so each chunk URL fits the byte limit. An already-fitting request is a one-step plan, unless an - active :func:`parallel_chunks` block asks the plan to fan out more - finely. See the module docstring for the concurrency model. + active :func:`parallel_chunks` block requests a finer fan-out. See the + module docstring for the concurrency model. Parameters ---------- @@ -209,7 +204,7 @@ def multi_value_chunked( ------- Callable A *synchronous* wrapper ``wrapper(args, *, finalize=...) -> - (df, response)`` that executes the underlying plan transparently + (df, response)`` that executes the underlying plan over the decorated async fetcher. Raises @@ -237,10 +232,10 @@ def wrapper( finalize: _Finalize = _passthrough_result, ) -> tuple[pd.DataFrame, Any]: limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit - # Resolve the parallel_chunks dial ``n`` through the configuration + # Resolve the parallel_chunks setting ``n`` through the configuration # chain (1 = off unless a ``parallel_chunks``/``configure`` block or - # the config file raised it; otherwise the requested total chunk - # cap). It only affects *planning*, done here up front, so a later + # the config file set it higher; otherwise the requested total chunk + # cap). It affects only planning, done here before execution, so a later # resume — which re-issues the already-planned chunks — reuses this # plan rather than resolving again. plan = ChunkPlan( diff --git a/dataretrieval/ogc/dates.py b/dataretrieval/ogc/dates.py index 4db800ead..b3d9a3bc6 100644 --- a/dataretrieval/ogc/dates.py +++ b/dataretrieval/ogc/dates.py @@ -31,7 +31,7 @@ _DURATION_RE = re.compile(r"^[Pp]T?\d") -# OGC API parameters that carry a date/datetime value (single string, +# OGC API parameters that take a date/datetime value (single string, # two-element range, or interval/duration string) rather than a multi-value # string list. Used by ``_construct_api_requests`` to keep them out of the # POST/CQL2 multi-value path and to route them through ``_format_api_dates``, @@ -45,7 +45,7 @@ def _parse_datetime(value: str) -> datetime | None: """Parse a single datetime string against the supported formats. - Returns a ``datetime`` (tz-aware iff the input carried a UTC offset), + Returns a ``datetime`` (tz-aware iff the input included a UTC offset), or ``None`` if no format matched. """ # ``datetime.strptime`` accepts a numeric offset like ``+00:00`` but not @@ -75,7 +75,7 @@ def _format_one(dt: str | None, *, date: bool) -> str | None: return parsed.strftime("%Y-%m-%d") # Naive inputs are interpreted in the system local zone (for backwards # compatibility). Use ``.astimezone()`` rather than a fixed offset so each - # value is resolved against the DST rules for ITS OWN date — a frozen + # value is resolved against the DST rules for its own date — a fixed # ``datetime.now()`` offset shifted off-season inputs by an hour. aware = parsed if parsed.tzinfo is not None else parsed.astimezone() return aware.astimezone(ZoneInfo("UTC")).strftime("%Y-%m-%dT%H:%M:%SZ") @@ -134,13 +134,13 @@ def _format_api_dates( The caller's own spelling of this argument, used as the subject of every message raised here. Defaults to a generic "date input"; pass the real parameter name (``"time"``, ``"last_modified"``) so a caller - correcting the error edits an argument their getter actually accepts. + correcting the error edits an argument their getter accepts. single_value_hint : str, optional How the "too many values" message describes an acceptable single value. Wording only -- a getter that rejects some of the default's - forms (``get_ratings`` refuses durations) enforces that itself and + forms (``get_ratings`` rejects durations) enforces that itself and passes a hint naming only what it accepts, so the remedy does not - send a caller straight into its rejection. + direct a caller to a value that is rejected. Returns ------- diff --git a/dataretrieval/ogc/engine.py b/dataretrieval/ogc/engine.py index b26c49170..0694fc4a5 100644 --- a/dataretrieval/ogc/engine.py +++ b/dataretrieval/ogc/engine.py @@ -2,22 +2,22 @@ This module holds OGC API Features orchestration — OGC cursor/response strategies and the chunked fetch entry point :func:`get_ogc_data`. Generic -pagination and sync dispatch live in :mod:`dataretrieval.transport`; request -construction lives in :mod:`~dataretrieval.ogc.requests`. The surrounding -concerns live in sibling modules this one composes, each with its own reason +pagination and sync dispatch are in :mod:`dataretrieval.transport`; request +construction is in :mod:`~dataretrieval.ogc.requests`. The surrounding +concerns are in sibling modules this one composes, each with its own reason to change: :mod:`~dataretrieval.ogc.dates` (time-parameter marshalling), :mod:`~dataretrieval.ogc.errors` (HTTP error mapping), and :mod:`~dataretrieval.ogc.shaping` (GeoJSON features to DataFrame and result finalization). It is deliberately free of any Water-Data-specific constants -so a sibling package (e.g. NGWMN) can drive it without importing +so a sibling package (e.g. NGWMN) can use it without importing ``dataretrieval.waterdata``. API-specific behavior is supplied by the caller: * ``output_id`` — the user-facing column the wire ``id`` is renamed to, - passed explicitly (no collection map lives here). + passed explicitly (no collection map is defined here). * ``base_url`` — the OGC API base to target. -* ``extra_id_cols`` — synthetic id columns to push to the end of a result. +* ``extra_id_cols`` — synthetic id columns to move to the end of a result. * ``dialect`` — an :class:`OgcDialect` describing which collections need POST/CQL2 and which use date-only (vs. full datetime) time arguments. """ @@ -42,7 +42,7 @@ _require_positive_int, ) -# Request construction stays in its canonical module; the engine imports only +# Request construction is defined in its canonical module; the engine imports only # the symbols its orchestration uses. from dataretrieval.ogc.requests import ( _construct_api_requests, @@ -85,18 +85,18 @@ def _next_req_url( Notes ----- - - Returns None when the response carries no features. + - Returns None when the response contains no features. - Expects the response JSON to contain a "links" list with objects having "rel" and "href" keys. - Checks for the "next" relation in the "links" to determine the next URL. """ if body is None: body = resp.json() - # Stop paging when the response carries no features. Key off ``features`` + # Stop paging when the response contains no features. Key off ``features`` # rather than ``numberReturned``: the main Water Data API reports - # ``numberReturned`` but the NGWMN OGC API omits it, so trusting it would - # refuse to follow a ``next`` link on a page that actually carries - # features (mirrors the same guard in :func:`_get_resp_data`). + # ``numberReturned`` but the NGWMN OGC API omits it, so relying on it would + # stop following ``next`` links on a page that has + # features (the same check as in :func:`_get_resp_data`). if not (body.get("features") or []): return None for link in body.get("links", []): @@ -124,7 +124,7 @@ def _ogc_parse_response( ) -> tuple[pd.DataFrame, str | None]: """Parse one OGC API page: extract the DataFrame and the next-page URL. - The parse strategy :func:`_walk_pages` hands to + The parse strategy :func:`_walk_pages` passes to :func:`~dataretrieval.transport.pagination.paginate`. Coerces falsy cursors (empty href, etc.) to ``None`` so the paginate loop's ``while cursor is not None`` terminates instead of looping forever on an @@ -153,7 +153,7 @@ async def _walk_pages( """ Iterate paginated OGC API responses and aggregate them into one DataFrame. - Thin wrapper that hands off to + Thin wrapper that delegates to :func:`~dataretrieval.transport.pagination.paginate` with OGC-specific strategies: pages are parsed via :func:`_get_resp_data` (through :func:`_ogc_parse_response`) and the next-page cursor is the @@ -169,7 +169,7 @@ async def _walk_pages( req : httpx.Request The initial HTTP request to send. client : httpx.AsyncClient, optional - Caller-borrowed client; ``None`` defers client management to + Caller-supplied client; ``None`` leaves client management to :func:`~dataretrieval.transport.pagination.paginate`. row_cap : int, optional Stop following pages once this many rows have accumulated and @@ -182,9 +182,9 @@ async def _walk_pages( pd.DataFrame A DataFrame containing the aggregated results from all pages. httpx.Response - Aggregated response — initial-request URL (for query identity), - final page's headers (so downstream sees current rate-limit - state), and cumulative ``elapsed`` summed across pages. + Aggregated response — initial-request URL (for query identity), final page's + headers (so downstream code receives current rate-limit state), and cumulative + ``elapsed`` summed across pages. Raises ------ @@ -243,7 +243,7 @@ def get_ogc_data( ``"monitoring-locations"``, ``"continuous"``). output_id : str The user-facing id column the wire ``id`` is renamed to. Required — - the per-API collection-to-id map lives in the caller, not here. + the per-API collection-to-id map is the caller's, not this function's. max_rows : int, optional Stop paginating once this many rows have been collected and truncate the result to exactly ``max_rows``. ``None`` (default) @@ -258,7 +258,7 @@ def get_ogc_data( The adapter supplies this semantic fact; ``skip_geometry`` and geopandas availability then select the concrete frame representation. extra_id_cols : set or frozenset, optional - Synthetic id columns to push to the end of a result frame (see + Synthetic id columns to move to the end of a result frame (see :func:`_arrange_cols`). Defaults to an empty set. dialect : OgcDialect, optional Per-API request quirks (CQL2-only collections, date-only collections). @@ -268,8 +268,8 @@ def get_ogc_data( building the query from ``args``. With a body, only the ``properties``, ``bbox``, ``limit``, ``skip_geometry``, and ``convert_type`` keys of ``args`` are consulted; there are no - multi-value axes to chunk, and the body's size is the server's - judgement, mirroring the planner's cql-json passthrough. + multi-value axes to chunk, and the body's size is for the server + to accept or reject, as with the planner's cql-json passthrough. Returns ------- @@ -284,10 +284,10 @@ def get_ogc_data( - Handles optional arguments such as `convert_type`. - Applies column cleanup and reordering based on collection and properties. """ - # Enforce a genuine positive integer up front: a float (even ``10.0``) or - # ``bool`` would pass a bare ``< 1`` check and then crash deep in - # ``pd.DataFrame.head`` with an opaque ``TypeError`` after HTTP I/O has - # already fired. Shared with ``parallel_chunks(n)`` via the helper. + # Enforce a positive integer before any request: a float (even ``10.0``) or + # ``bool`` would pass a bare ``< 1`` check and then raise an unclear + # ``TypeError`` from ``pd.DataFrame.head`` after the HTTP requests have + # already been sent. Shared with ``parallel_chunks(n)`` via the helper. if max_rows is not None: _require_positive_int(max_rows, "max_rows") @@ -296,8 +296,8 @@ def get_ogc_data( args = args.copy() args["collection"] = collection args = _switch_arg_id(args, id_name=output_id, collection=collection) - # Capture `properties` before the id-switch so post-processing sees - # the user-facing names, not the wire-format ones. + # Capture `properties` before the id-switch so post-processing receives the + # user-facing names, not the wire-format ones. properties = args.get("properties") args["properties"] = _switch_properties_id( properties, id_name=output_id, collection=collection @@ -307,14 +307,14 @@ def get_ogc_data( # Choose one semantic frame shape for the whole request. Every page and # the all-empty finalizer receive these same values, so frame type never - # depends on whether a particular page happens to carry geometry. + # depends on whether a particular page includes geometry. include_geometry = spatial and not bool(args.get("skip_geometry", False)) geopd = GEOPANDAS and include_geometry # Post-processing is injected into the chunker rather than applied here, - # so it runs on *every* exit: the normal return AND a later + # so it runs on every exit: the normal return and a later # ``exc.call.resume()`` after a ChunkInterrupted (which never re-enters - # this function). ``_finalize_ogc`` is the single source of result shape; + # this function). ``_finalize_ogc`` is the one place result shape is applied; # it also applies ``max_rows`` to the *combined* frame so the cap is the # exact total even when the plan chunks or the call is resumed, while # the per-chunk ``row_cap`` bound below only early-stops each chunk's @@ -336,7 +336,7 @@ def get_ogc_data( if cql_body is not None: # ``args["properties"]`` holds the wire property list after the # id-switch above; ``finalize`` holds the pre-switch, user-facing - # list, exactly as on the chunked path. + # list, as on the chunked path. req = _construct_cql_request( collection, cql_body, @@ -370,7 +370,7 @@ def get_ogc_data( # same way ``finalize`` binds its own state: with ``functools.partial``. # The plan sizes candidate chunks and a later ``exc.call.resume()`` # rebuilds them through these same bound callables, so the values the - # call was created with reach every chunk — even a resume fired long + # call was created with apply to every chunk — even a resume invoked long # after this function returned — without any ambient state to snapshot. build_request = functools.partial( _construct_api_requests, base_url=base_url, dialect=dialect @@ -409,7 +409,7 @@ async def _fetch_once( the decorator passes args through unchanged. The decorator gathers every chunk over one shared :class:`httpx.AsyncClient` (concurrency bounded by a semaphore, sized from the effective ``concurrency`` - setting) and returns a *synchronous* wrapper, so ``get_ogc_data`` drives + setting) and returns a *synchronous* wrapper, so ``get_ogc_data`` calls it synchronously. The return shape is ``(frame, response)``. """ req = build_request(**args) diff --git a/dataretrieval/ogc/errors.py b/dataretrieval/ogc/errors.py index 326d033b8..a9b7b021c 100644 --- a/dataretrieval/ogc/errors.py +++ b/dataretrieval/ogc/errors.py @@ -28,12 +28,12 @@ def _error_body(resp: httpx.Response) -> str: str An error message string assembled per status code: - * **429** — predefined message describing the rate-limit and pointing - at the API-token path; the response body is not consulted. + * **429** — predefined message describing the rate-limit and naming the + API-token option; the response body is not consulted. * **every other status** — a supported JSON error body (the USGS ``code``/``description`` envelope or a gateway ``message``) when present; otherwise ``": . "`` with the first - 200 characters of ``resp.text``; an empty body degrades to + 200 characters of ``resp.text``; an empty body yields ``": ."``, except **403**, which falls back to :data:`_FORBIDDEN_CAUSES` so a credential problem is named. @@ -59,7 +59,7 @@ def _error_body(resp: httpx.Response) -> str: #: What a 403 means when the service sends no error envelope. Both causes are -#: named because the credential one is far more common and was omitted. +#: named because the credential one is more common and was omitted. _FORBIDDEN_CAUSES = ( "Query request denied. The API key may be missing, expired, or revoked " "(see API_USGS_PAT), or the query may exceed server limits." @@ -129,8 +129,8 @@ def _raise_for_non_200(resp: httpx.Response) -> None: transient types (:class:`~dataretrieval.exceptions.TransientError`) are distinguished so ``ChunkedCall`` can wrap them as a resumable :class:`~dataretrieval.interruptions.QuotaExhausted` / - :class:`~dataretrieval.interruptions.ServiceInterrupted`. The - chunker won't resume a fatal + :class:`~dataretrieval.interruptions.ServiceInterrupted`. The executor does not + resume a fatal :class:`~dataretrieval.exceptions.HTTPError` (not a ``TransientError``). """ status = resp.status_code diff --git a/dataretrieval/ogc/filters.py b/dataretrieval/ogc/filters.py index 2d4bc451d..8550508d2 100644 --- a/dataretrieval/ogc/filters.py +++ b/dataretrieval/ogc/filters.py @@ -7,7 +7,7 @@ Internal helpers used by ``chunking.multi_value_chunked``'s joint planner: ``_split_top_level_or`` (clause partitioning), ``_is_chunkable`` (filter-language gate), and -``_check_numeric_filter_pitfall`` (the lexicographic-comparison guard). +``_check_numeric_filter_pitfall`` (the lexicographic-comparison check). ``_quote_cql_str`` escapes a single CQL-text string literal, shared by any getter that *builds* a CQL filter (e.g. ``waterdata.ratings``). @@ -57,7 +57,7 @@ def _quote_cql_str(value: str) -> str: CQL2 text escapes a ``'`` inside a string literal by doubling it, so ``O'Brien`` becomes ``O''Brien`` (wrap the result in ``'…'`` at the call - site). Defends against malformed filters / injection on arbitrary user + site). Prevents malformed filters and injection from arbitrary user input. Shared by every getter that builds a CQL-text literal (e.g. the STAC ``/search`` filter in ``waterdata.ratings``). """ @@ -74,10 +74,10 @@ def _skip_space(expr: str, i: int) -> int: def _resume_after_or(expr: str, i: int) -> int | None: """Where the clause after a top-level ``OR`` begins, if one starts at ``i``. - ``i`` is the index of a space that may open a ``OR`` - separator. Returns the index of the next clause's first character, or - ``None`` when this space does not begin one -- so the caller's test is - "is this a separator?" rather than four nested boundary checks. + ``i`` is the index of a space that may open a ``OR`` separator. + Returns the index of the next clause's first character, or ``None`` when this space + does not begin one -- so the caller tests one condition rather than four nested + boundary checks. The trailing space is required: without it ``A ORDER BY b`` would split on the ``OR`` inside ``ORDER``. @@ -94,8 +94,8 @@ def _resume_after_or(expr: str, i: int) -> int | None: def _skip_quoted(expr: str, i: int) -> int: """Index just past the quoted span opening at ``i``. - An unterminated quote swallows the rest of the expression. A doubled - ``''`` escape reads as close-then-reopen, which nets to the same state. + An unterminated quote extends to the end of the expression. A doubled + ``''`` escape reads as close-then-reopen, which produces the same state. """ close = expr.find(expr[i], i + 1) return len(expr) if close == -1 else close + 1 @@ -104,8 +104,8 @@ def _skip_quoted(expr: str, i: int) -> int: def _iter_top_level_spaces(expr: str) -> Iterator[int]: """Yield the indices of whitespace outside quotes and parens. - Owns the depth/quote state so callers only see split candidates; quoted - spans are skipped wholesale. + Owns the depth/quote state so callers receive only split candidates; quoted + spans are skipped entirely. """ depth = 0 i = 0 @@ -127,7 +127,7 @@ def _iter_top_level_spaces(expr: str) -> Iterator[int]: def _split_top_level_or(expr: str) -> list[str]: """Split ``expr`` at each top-level ``OR``, respecting quotes and parens. - ``OR`` tokens inside ``(A OR B)`` or ``'word OR word'`` are left alone. + ``OR`` tokens inside ``(A OR B)`` or ``'word OR word'`` are not split. Matching is case-insensitive; whitespace around each part is stripped; empty parts are dropped. """ @@ -153,7 +153,7 @@ def _numeric_pitfall_error(field: str, offense: str) -> ValueError: f"literals with HTTP 500; even quoting the literal gives a " f"lexicographic comparison (``value > '10'`` matches " f"``value='34.52'``, ``parameter_code = '60'`` matches nothing " - f"because the real codes are ``'00060'``-shaped). For a true " + f"because the real codes are of the form ``'00060'``). For a true " f"numeric filter, fetch a wider result and reduce in pandas." ) @@ -198,14 +198,14 @@ def _check_numeric_filter_pitfall(filter_expr: str) -> None: ``hydrologic_unit_code``, ``channel_flow``). Any unquoted numeric comparison — ``value >= 1000``, ``parameter_code = 60``, ``parameter_code IN (60, 61)``, ``value BETWEEN 5 AND 10`` — either gets - rejected with HTTP 500 or silently produces lexicographic results. + rejected with HTTP 500 or produces lexicographic results without an error. Zero-padded codes are the worst case (``parameter_code = '60'`` matches - nothing because the real codes are ``'00060'``-shaped). + nothing because the real codes are of the form ``'00060'``). Quoted literals (``value >= '1000'``) are not flagged — the caller has - signalled they know the column is textual. + indicated that the column is textual. """ - # Mask quoted strings so ``name = 'value > 5'`` doesn't false-positive. + # Mask quoted strings so ``name = 'value > 5'`` is not flagged. masked = ( _QUOTED_STR_RE.sub("''", filter_expr) if "'" in filter_expr else filter_expr ) diff --git a/dataretrieval/ogc/interruptions.py b/dataretrieval/ogc/interruptions.py index 3b76f12ee..4e3bd30a0 100644 --- a/dataretrieval/ogc/interruptions.py +++ b/dataretrieval/ogc/interruptions.py @@ -1,19 +1,19 @@ """Deprecated alias for :mod:`dataretrieval.interruptions`. -The classes live in :mod:`dataretrieval.interruptions`, where the base class is +The classes are defined in :mod:`dataretrieval.interruptions`, where the base class is named :class:`~dataretrieval.interruptions.FanOutInterrupted`. That move, and ``ChunkInterrupted`` staying a permanent alias rather than a shim, are ADR 0008. This path is the one v1.2.0 published, when the classes were defined here. Importing this module emits a :class:`DeprecationWarning` and re-exports the -taxonomy. The re-exported objects are the *same objects*, not copies, so +taxonomy. The re-exported objects are the same objects, not copies, so ``ogc.interruptions.ChunkInterrupted is dataretrieval.ChunkInterrupted`` and ``except`` clauses behave identically through either spelling. Only the module *path* is deprecated. -``dataretrieval.ogc.__init__`` deliberately does not import this module, so -``import dataretrieval`` and ``import dataretrieval.ogc`` stay silent. The -warning fires only for code that names ``ogc.interruptions`` itself. +``dataretrieval.ogc.__init__`` deliberately does not import this module, so ``import +dataretrieval`` and ``import dataretrieval.ogc`` emit no warning. The warning is emitted +only for code that imports ``ogc.interruptions`` itself. """ from __future__ import annotations @@ -27,7 +27,7 @@ ) #: When the alias may be deleted. Read from the shared horizon table rather -#: than spelled here, so it is audited and bumped with every other published +#: than spelled here, so it is reviewed and extended with every other published #: removal. OGC_INTERRUPTIONS_REMOVAL_DATE = REMOVALS["ogc.interruptions"] @@ -44,9 +44,9 @@ "(`from dataretrieval import ChunkInterrupted`)", removal=OGC_INTERRUPTIONS_REMOVAL_DATE, detail="The exception classes are unchanged and are the same objects; " - "only this import path is going away. Removal is planned for a future " + "only this import path is being removed. Removal is planned for a future " "major release.", - # 1 lands the warning on the line that imported this module -- an import - # has no deeper user frame to point at. + # 1 attributes the warning to the line that imported this module -- an import + # has no deeper user frame to attribute it to. stacklevel=1, ) diff --git a/dataretrieval/ogc/planning.py b/dataretrieval/ogc/planning.py index c4d28f476..0f7ae0457 100644 --- a/dataretrieval/ogc/planning.py +++ b/dataretrieval/ogc/planning.py @@ -3,16 +3,16 @@ This module holds the side-effect-free planning half of the chunker: deciding how to split one over-budget OGC request into URL-fitting chunks (:class:`ChunkPlan` and the axis/byte-accounting helpers). -It has no event loop, retry policy, or network state — those live in +It has no event loop, retry policy, or network state — those are in :mod:`dataretrieval.ogc.chunking` (resumable execution) and :mod:`dataretrieval.transport.retry` (retry policy), which import the plan and -drive it. +execute it. Result recombination — reassembling the per-chunk frames and responses back into one result (:func:`~dataretrieval.combining._combine_chunk_frames`, :func:`~dataretrieval.combining._combine_chunk_responses`, etc.) — -lives in the top-level :mod:`dataretrieval.combining` module. +is in the top-level :mod:`dataretrieval.combining` module. """ from __future__ import annotations @@ -34,13 +34,12 @@ _split_top_level_or, ) -# Any list-shaped kwarg with >1 element is chunked (comma-joined per -# sub-list in the URL); ~90 OGC params qualify, so we denylist the few -# exceptions rather than maintain a growing allowlist. Excluded because: -# ``properties`` defines the column schema; ``bbox`` is a fixed coord -# tuple; date/time params are intervals, not enumerable sets; ``filter`` -# is handled as its own OR-axis in ``_extract_axes``; and ``limit`` / -# ``skip_geometry`` / ``filter_lang`` are scalar by contract. +# Any list-shaped kwarg with >1 element is chunked (comma-joined per sub-list in the +# URL); ~90 OGC params qualify, so the few exceptions are denylisted rather than +# maintaining a growing allowlist. Excluded because: ``properties`` defines the column +# schema; ``bbox`` is a fixed coord tuple; date/time params are intervals, not +# enumerable sets; ``filter`` is handled as its own OR-axis in ``_extract_axes``; and +# ``limit`` / ``skip_geometry`` / ``filter_lang`` are scalar by contract. _NEVER_CHUNK = frozenset( { "properties", @@ -99,9 +98,9 @@ def _try_build( """Attempt to construct a request, returning ``None`` on overflow. ``httpx.URL`` enforces a hard 64 KB cap per URL component and raises - ``httpx.InvalidURL`` for anything bigger. Both :func:`_safe_request_bytes` - and :class:`ChunkPlan`'s initial-request probe need exactly this - "build-or-None" step, so it lives here once. + ``httpx.InvalidURL`` for anything bigger. Both :func:`_safe_request_bytes` and + :class:`ChunkPlan`'s initial-request probe need this build-or-None step, so it is + defined here once. Parameters ---------- @@ -132,9 +131,9 @@ def _safe_request_bytes( ``httpx.URL`` enforces a hard 64 KB cap per URL component (``MAX_URL_LENGTH``) and raises ``httpx.InvalidURL`` for anything - bigger. We report ``url_limit + 1`` on overflow so the greedy + bigger. ``url_limit + 1`` is reported on overflow so the greedy halving loop in :meth:`ChunkPlan._plan` keeps shrinking the - largest axis until ``httpx.Request`` can be constructed at all. + largest axis until ``httpx.Request`` can be constructed. Parameters ---------- @@ -165,7 +164,7 @@ def _check_unchunkable_request( Passthrough when the single request fits or when the filter is in a language the chunker doesn't manage (cql-json) — the server, not the - chunker, judges that request. Raises + chunker, accepts or rejects that request. Raises :class:`~dataretrieval.exceptions.Unchunkable` when the request is over budget and has nothing to split. """ @@ -219,7 +218,7 @@ def chunk_bytes(self, chunk: list[str]) -> int: """ Return the URL-encoded byte count this chunk contributes to the request. - ``quote_plus`` is faithful to what the real URL builder + ``quote_plus`` matches what the real URL builder produces, so values containing characters that expand under URL encoding (``%``, ``+``, ``/``, ``&``, …) can't be mis-ranked. @@ -239,9 +238,8 @@ def render(self, chunk: list[str]) -> Any: """ Convert a chunk into the form the URL builder expects. - List axes yield a fresh list of atoms (``build_request`` will - comma-join); the filter axis yields a pre-joined string (CQL - doesn't take a list). + List axes yield a new list of atoms (``build_request`` will comma-join); the + filter axis yields a pre-joined string (CQL does not accept a list). Parameters ---------- @@ -312,12 +310,12 @@ def _extract_axes(args: dict[str, Any]) -> list[_Axis]: def _split_at(chunks: list[list[str]], idx: int) -> None: """Replace ``chunks[idx]`` in place with its two contiguous halves. - The single primitive both planning passes use to fan an axis out. It + The single primitive both planning passes use to split an axis. It preserves the partition invariants every consumer relies on: *coverage* - (each atom survives, exactly once) and *contiguous, deterministic order* + (each atom appears exactly once) and *contiguous, deterministic order* (resume and :meth:`ChunkPlan.iter_chunk_args` depend on it). Kept in one place so those invariants can't drift between :meth:`ChunkPlan._plan` - (byte-driven) and :meth:`ChunkPlan._refine` (fan-out-driven). + (by bytes) and :meth:`ChunkPlan._refine` (by chunk count). """ chunk = chunks[idx] mid = len(chunk) // 2 @@ -328,7 +326,7 @@ class ChunkPlan: """ Strategy for issuing one user-level request as URL-fitting chunks. - Every chunk URL fits ``url_limit``. Constructing a plan *is* planning: + Every chunk URL fits ``url_limit``. Constructing a plan performs the planning: ``ChunkPlan(args, build_request, url_limit)`` extracts the chunkable axes, runs greedy halving on the biggest chunk across all axes, and stores the result. @@ -350,16 +348,14 @@ class ChunkPlan: Byte budget for the request (URL + body) — a hard ceiling every chunk must fit. max_chunks : int, optional - Hard cap on the plan's total chunk count (default ``1`` = off). - ``1`` chunks only as much as ``url_limit`` requires — the most - conservative plan, fewest chunks — so a fitting request is a - passthrough. A cap of ``2`` or more fans the plan out to up to - ``max_chunks`` chunks overall (the cartesian product across axes, - never fewer than the byte budget already forces). The cap applies to - the plan as a whole, not per axis, so several multi-value axes can't - multiply past it. The plan never exceeds the cap and may land below it - when no whole split lands on it exactly. ``max_chunks`` is a - chunk count, so a value below ``1`` (``0`` or negative) is a + Hard cap on the plan's total chunk count (default ``1`` = off). ``1`` chunks + only as much as ``url_limit`` requires — the plan with the fewest chunks — so a + fitting request is a passthrough. A cap of ``2`` or more fans the plan out to up + to ``max_chunks`` chunks overall (the cartesian product across axes, never fewer + than the byte budget already forces). The cap applies to the plan as a whole, + not per axis, so several multi-value axes can't multiply past it. The plan never + exceeds the cap and may fall below it when no whole split reaches it exactly. + ``max_chunks`` is a chunk count, so a value below ``1`` (``0`` or negative) is a caller error and raises ``ValueError``. Set from the :func:`~dataretrieval.ogc.chunking.parallel_chunks` ``n``; see :meth:`_refine`. @@ -414,7 +410,7 @@ def __init__( return # When the un-chunked URL builds, preserve it as ``canonical_url`` so - # ``BaseMetadata.url`` echoes the user's original query verbatim. + # ``BaseMetadata.url`` reports the user's original query verbatim. initial_request = _try_build(build_request, args) fits = False if initial_request is not None: @@ -444,7 +440,7 @@ def _plan( Halving continues until the worst-case chunk URL fits ``url_limit``, mutating ``self.chunks`` in place. List axes and the - filter axis are treated uniformly — each is just a list of atoms + filter axis are treated uniformly — each is a list of atoms joined by its axis's separator. Raises @@ -489,24 +485,22 @@ def _refine(self, max_chunks: int) -> None: """ Fan the plan out more finely than the byte budget alone requires. - This is the ``parallel_chunks`` dial: see - :func:`~dataretrieval.ogc.chunking.parallel_chunks` for why a caller - would want this, and :class:`ChunkPlan`'s ``max_chunks`` parameter for + This is what ``parallel_chunks`` controls: see + :func:`~dataretrieval.ogc.chunking.parallel_chunks` for when a caller + uses this, and :class:`ChunkPlan`'s ``max_chunks`` parameter for the cap's contract (total-not-per-axis, a hard ceiling that may land below the cap). - Implementation. Each split multiplies the plan by ``(k+1)/k`` for the - chosen axis (adding ``total // k`` chunks, not one), so a split - is taken only when it keeps :attr:`total` within the cap. When no - in-budget split remains, the plan stops *below* the cap rather than - overshooting (two even axes can reach 4 but not 5, so a cap of 5 yields - 4). Each split picks the single largest splittable chunk among the - in-budget axes (ties broken by axis-extraction order, then lowest - index), so growth is distributed round-robin rather than one axis - saturating before another is touched. Purely additive — only ever - *splits* existing chunks, so the byte pass's work and the ``url_limit`` - invariant are both preserved, and it never raises. A no-op at - ``max_chunks == 1``. + Implementation. Each split multiplies the plan by ``(k+1)/k`` for the chosen + axis (adding ``total // k`` chunks, not one), so a split is taken only when it + keeps :attr:`total` within the cap. When no in-budget split remains, the plan + stops below the cap rather than exceeding it (two even axes can reach 4 but not + 5, so a cap of 5 yields 4). Each split picks the single largest splittable chunk + among the in-budget axes (ties broken by axis-extraction order, then lowest + index), so growth is distributed round-robin rather than one axis saturating + before another is split. Additive only: it only splits existing chunks, so the + byte pass's work and the ``url_limit`` invariant are both preserved, and it + never raises. A no-op at ``max_chunks == 1``. Parameters ---------- @@ -566,7 +560,7 @@ def _best_refine_candidate( for axis in self.axes: axis_chunks = self.chunks[axis.arg_key] if total + total // len(axis_chunks) > max_chunks: - continue # any split of this axis would overshoot the cap + continue # any split of this axis would exceed the cap axis_best, axis_best_size = self._largest_chunk_in(axis_chunks) if axis_best_size > candidate_size: candidate, candidate_size = (axis, axis_best), axis_best_size diff --git a/dataretrieval/ogc/policy.py b/dataretrieval/ogc/policy.py index 0959e682a..c92c7da1a 100644 --- a/dataretrieval/ogc/policy.py +++ b/dataretrieval/ogc/policy.py @@ -1,15 +1,15 @@ """Low-level OGC policy: the dialect type and control validation. -This module is the single source of truth for the :class:`OgcDialect` type +This module is the one definition of the :class:`OgcDialect` type (per-API quirks the generic request builder needs) and OGC control validation. It depends only on the stdlib, so any OGC submodule can import it without creating cycles. It names no endpoint: which API an OGC call targets is the *adapter's* -policy, supplied per call as ``base_url``. A default here would quietly -point every generic OGC caller at one API. +policy, supplied per call as ``base_url``. A default here would +direct every generic OGC caller to one API. -It must NOT import engine, shaping, or any collection adapter. +It must not import engine, shaping, or any collection adapter. """ from __future__ import annotations @@ -38,7 +38,7 @@ def _require_positive_int( @dataclass(frozen=True) class OgcDialect: - """Per-API quirks the generic request builder needs to know about. + """Per-API differences the generic request builder must handle. Attributes ---------- @@ -51,9 +51,9 @@ class OgcDialect: ``last_modified`` parameter is always rendered as a full datetime regardless of this set. time_cols : frozenset[str] - Result columns to coerce to datetime when ``convert_type`` is set. - Empty by default, so the generic engine carries no API-specific - column knowledge; each API supplies its own. + Result columns to coerce to datetime when ``convert_type`` is set. Empty by + default, so the generic engine holds no API-specific column list; each API + supplies its own. numerical_cols : frozenset[str] Result columns to coerce to numeric when ``convert_type`` is set. sort_cols : tuple[str, ...] diff --git a/dataretrieval/ogc/requests.py b/dataretrieval/ogc/requests.py index 151fb50a6..1427dc387 100644 --- a/dataretrieval/ogc/requests.py +++ b/dataretrieval/ogc/requests.py @@ -1,8 +1,9 @@ """OGC argument normalization and HTTP request construction. The API to target and its quirks are explicit parameters (``base_url``, -``dialect``) -- construction states everything it needs. Queryables and schema -execution live in :mod:`dataretrieval.ogc.schema` and are not re-exported here, +``dialect``) -- construction takes everything it needs as parameters. +Queryables and schema execution are in :mod:`dataretrieval.ogc.schema` and are +not re-exported here, so this module does not depend on the part of OGC that executes HTTP (ADR 0007). """ @@ -25,7 +26,7 @@ # ``AGENCY-ID``: a hyphen-separated agency prefix and local id. The local id # may itself contain hyphens (``\S+`` after the first separator) — NGWMN -# aggregates many non-USGS agencies whose local ids aren't bare digits, so +# aggregates many non-USGS agencies whose local ids are not only digits, so # only the agency prefix is constrained to be hyphen/space-free. _MONITORING_LOCATION_ID_RE = re.compile(r"[^-\s]+-\S+") @@ -37,7 +38,7 @@ def _switch_arg_id(ls: dict[str, Any], id_name: str, collection: str) -> dict[str, Any]: """Switch argument id from its package-specific identifier to the - standardized "id" key that the API recognizes.""" + standardized "id" key that the API accepts.""" collection_id = collection.replace("-", "_") + "_id" if "id" not in ls: if collection_id in ls: @@ -82,7 +83,7 @@ def _ogc_query_params( limit: int | None, skip_geometry: bool | None, ) -> dict[str, Any]: - """Add the shared OGC query knobs to ``params`` (mutated in place).""" + """Add the shared OGC query parameters to ``params`` (mutated in place).""" if skip_geometry is not None: params["skipGeometry"] = skip_geometry params["limit"] = 50000 if limit is None or limit > 50000 else limit @@ -101,7 +102,9 @@ def _is_post_param(value: Any) -> bool: def _partition_cql2( params: dict[str, Any], ) -> tuple[dict[str, Any], dict[str, Any]]: - """CQL2 path: multi-value lists go to POST body, rest stay as URL params.""" + """CQL2 path: multi-value lists are placed in the POST body, the rest remain URL + params. + """ post_params = {key: value for key, value in params.items() if _is_post_param(value)} url_params = {key: value for key, value in params.items() if key not in post_params} return url_params, post_params @@ -167,7 +170,7 @@ def _construct_api_requests( """Construct an HTTP request object for the specified OGC API collection. ``base_url`` is required: this package is API-neutral and names no API of - its own, so the adapter naming the collection states the API it targets. + its own, so the adapter naming the collection also names the API it targets. ``dialect`` defaults to a plain OGC API with no per-collection quirks. """ service_url = _items_url(collection, base_url) @@ -239,9 +242,9 @@ def _construct_cql_request( # Argument normalization helpers # --------------------------------------------------------------------------- -# Iterable-shaped params that ``_get_args`` must NOT push through -# ``_normalize_str_iterable`` (date-range params may carry ``pd.NaT``/None or -# interval strings; ``bbox`` is ``list[float]``). Every OGC caller gets these; +# Iterable params that ``_get_args`` must not pass through +# ``_normalize_str_iterable`` (date-range params may contain ``pd.NaT``/None or +# interval strings; ``bbox`` is ``list[float]``). These apply to every OGC caller; # an adapter with extra numeric params names only its extras via # ``prepare_request_args(..., extra_no_normalize=...)``. _NO_NORMALIZE_PARAMS = _DATE_RANGE_PARAMS | {"bbox"} @@ -335,14 +338,14 @@ def prepare_request_args( ) -> dict[str, Any]: """Build OGC request kwargs from a getter's ``locals()``. - Internal bookkeeping keys, caller-supplied exclusions, and ``None`` values + Internal control keys, caller-supplied exclusions, and ``None`` values are omitted. Identifiers and properties are validated; other iterables are normalized unless exempted. ``extra_no_normalize`` *adds* to the engine's own :data:`_NO_NORMALIZE_PARAMS` rather than replacing it, so an adapter names - only the params it owns and cannot silently drop the date-range exemptions - by forgetting to union them back in. + only the params it owns and cannot drop the date-range exemptions + by omitting them. """ no_normalize = _NO_NORMALIZE_PARAMS | frozenset(extra_no_normalize) to_exclude = {"collection", "service", "output_id"} diff --git a/dataretrieval/ogc/schema.py b/dataretrieval/ogc/schema.py index f85b170a0..5029d76f1 100644 --- a/dataretrieval/ogc/schema.py +++ b/dataretrieval/ogc/schema.py @@ -1,8 +1,8 @@ -"""Asking an OGC collection to describe itself. +"""Requests that describe an OGC collection. Queryables and collection schemas: which properties a collection accepts, and -what columns it returns. Separate from request construction because answering -these questions means *issuing* a request, and building one must not. +what columns it returns. Separate from request construction because these +require *issuing* a request, and request construction must not. """ from __future__ import annotations @@ -25,7 +25,7 @@ def _check_ogc_requests( ) -> tuple[dict[str, Any], httpx.Response]: """Retrieve one collection's queryables or response schema. - ``base_url`` names the API to ask; it defaults to the one in scope for the + ``base_url`` names the API to query; it defaults to the one in scope for the current call rather than to any particular collection. """ require_one_of(req_type, ("queryables", "schema"), name="req_type") @@ -41,9 +41,9 @@ def queryables_frame( """Tabulate one collection's queryable properties. Reading an OGC queryables document is protocol knowledge, not collection - knowledge, so it lives here rather than in any one API's getters -- every - OGC adapter in the package can offer the same table. ``base_url`` names - the API to ask, defaulting to the one in scope for the current call. + knowledge, so it is defined here rather than in any one API's getters -- every + OGC adapter in the package can return the same table. ``base_url`` names + the API to query, defaulting to the one in scope for the current call. Returns ------- diff --git a/dataretrieval/ogc/shaping.py b/dataretrieval/ogc/shaping.py index f59f4ba10..7486655e6 100644 --- a/dataretrieval/ogc/shaping.py +++ b/dataretrieval/ogc/shaping.py @@ -5,7 +5,7 @@ per the API dialect, the wire ``id`` rename + column ordering, row sort, ``max_rows`` truncation, and wrapping as ``BaseMetadata``. These are output conventions, with their own reason to change independent of the request and -pagination machinery in :mod:`dataretrieval.ogc.engine`. +pagination code in :mod:`dataretrieval.ogc.engine`. """ from __future__ import annotations @@ -31,7 +31,7 @@ # Water Data OGC coordinates are published in WGS84, so attach that CRS where the # GeoDataFrame is built (otherwise the result is CRS-naive and ``to_crs`` / -# spatial joins fail). Mirrors the constants in ``nldi`` (EPSG:4326) and ``nwis`` +# spatial joins fail). Matches the constants in ``nldi`` (EPSG:4326) and ``nwis`` # (EPSG:4269). _CRS = "EPSG:4326" @@ -53,9 +53,9 @@ def _empty_feature_frame( ``geopd`` and ``include_geometry`` are selected once before pagination and reused for every page and the final all-empty result. This keeps empty and - non-empty pages in the same frame family, so ordinary ``pd.concat`` is + non-empty pages in the same frame type, so ordinary ``pd.concat`` is sufficient. ``columns`` is supplied only when finalization has fetched the - collection schema; page-level empties intentionally remain schema-light. + collection schema; page-level empty frames intentionally have no schema columns. """ result_columns = list(columns or []) if not include_geometry: @@ -73,9 +73,8 @@ def _empty_feature_frame( def _attach_coordinates(df: pd.DataFrame, features: list[dict[str, Any]]) -> None: - """Attach a ``geometry`` column of raw coordinate lists (in place) when - any feature carries geometry. Shared by the non-geopandas GeoJSON - feature-frame builders. + """Attach a ``geometry`` column of raw coordinate lists (in place) when any feature + includes geometry. Shared by the non-geopandas GeoJSON feature-frame builders. """ geoms = [(f.get("geometry") or {}).get("coordinates") for f in features] if any(g is not None for g in geoms): @@ -83,15 +82,15 @@ def _attach_coordinates(df: pd.DataFrame, features: list[dict[str, Any]]) -> Non def _geo_feature_frame(features: list[dict[str, Any]]) -> pd.DataFrame: - """Build a ``GeoDataFrame`` from GeoJSON features, tolerating a missing + """Build a ``GeoDataFrame`` from GeoJSON features, accepting a missing ``geometry`` key. ``GeoDataFrame.from_features`` indexes ``feature["geometry"]`` directly, so collections that omit it (NGWMN observation collections, Water Data statistics features) would raise ``KeyError``. Default the key to ``None`` - for only those features, so features that already carry geometry (the + for only those features, so features that already include geometry (the common sites case) are passed through without a per-feature dict copy. - The single home for this upstream-schema workaround. + The one place this upstream-schema workaround is applied. """ return gpd.GeoDataFrame.from_features( [f if "geometry" in f else {**f, "geometry": None} for f in features], @@ -147,12 +146,10 @@ def _get_resp_data( Returns ------- gpd.GeoDataFrame or pd.DataFrame - A ``GeoDataFrame`` when ``geopd`` is True; otherwise a plain - ``DataFrame`` carrying the feature properties plus an ``id`` - column (always present, possibly all-None) and a ``geometry`` - column (coordinates list) when at least one feature includes - geometry. Returns an empty ``DataFrame`` when no features are - returned. + A ``GeoDataFrame`` when ``geopd`` is True; otherwise a plain ``DataFrame`` + holding the feature properties plus an ``id`` column (always present, possibly + all-None) and a ``geometry`` column (coordinates list) when at least one feature + includes geometry. Returns an empty ``DataFrame`` when no features are returned. Notes ----- @@ -168,12 +165,12 @@ def _get_resp_data( body = resp.json() # Key the empty-result short-circuit off ``features`` rather than # ``numberReturned``: the main Water Data API reports ``numberReturned``, - # but the NGWMN OGC API omits it, so trusting it would discard pages that - # actually carry features. An absent/empty ``features`` is also the real - # schema-drift shape (a 200 with no features) — treat it as empty rather - # than crash with a ``KeyError`` downstream, which ``_paginate`` would - # mistake for a transient transport error. The request-level mode gives - # this page the same frame family as every non-empty page in the walk. + # but the NGWMN OGC API omits it, so relying on it would discard pages that + # contain features. An absent/empty ``features`` is also the + # shape a schema change produces (a 200 with no features) — treat it as empty rather + # than raise a ``KeyError`` downstream, which ``_paginate`` would + # classify as a transient transport error. The request-level mode gives + # this page the same frame type as every non-empty page in the walk. features = body.get("features") or [] if not features: return _empty_feature_frame(geopd, include_geometry=include_geometry) @@ -195,9 +192,9 @@ def _deal_with_empty( ) -> pd.DataFrame: """Apply the collection schema when an entire result is empty. - Page-level empties already carry the request's frame family. The complete + Page-level empty frames already have the request's frame type. The complete column list is available only from explicit ``properties`` or the - collection schema, so construct the fully shaped empty once here rather + collection schema, so construct the fully shaped empty frame once here rather than fetching schema during pagination. """ if not return_list.empty: @@ -237,7 +234,7 @@ def _arrange_cols( output_id : str The name to which the 'id' column should be renamed if applicable. extra_id_cols : set or frozenset, optional - Synthetic, meaningless-to-user id columns to move to the end of the + Synthetic id columns that have no meaning to the user to move to the end of the result frame when the wire ``id`` is returned (i.e. ``properties`` was not specified). Defaults to an empty set (no reordering). @@ -251,7 +248,7 @@ def _arrange_cols( # Rename id column to output_id df = df.rename(columns={"id": output_id}) - # --- No explicit properties: move meaningless extra-id cols to end --- + # --- No explicit properties: move extra-id cols with no user meaning to the end --- if not properties or all(pd.isna(properties)): extra_id_col = set(df.columns).intersection(extra_id_cols) if extra_id_col: @@ -262,7 +259,7 @@ def _arrange_cols( return df # --- Explicit properties: select and reorder columns per the list --- - # Don't alias the caller's list — we mutate below. + # Do not alias the caller's list; it is mutated below. local_properties = list(properties) if "geometry" in df.columns and "geometry" not in local_properties: local_properties.append("geometry") @@ -283,8 +280,7 @@ def _type_cols(df: pd.DataFrame, dialect: OgcDialect) -> pd.DataFrame: The input DataFrame. dialect : OgcDialect Supplies ``time_cols`` / ``numerical_cols`` — which columns to - coerce to datetime/numeric. The engine itself holds no - API-specific column knowledge. + coerce to datetime/numeric. The engine itself holds no API-specific column list. Returns ------- @@ -307,8 +303,8 @@ def _sort_rows(df: pd.DataFrame, dialect: OgcDialect) -> pd.DataFrame: Sorting is applied only when the primary (first) sort column is present; any later sort columns also present become secondary keys. - This mirrors the historical Water Data behavior (sort by ``time``, - then ``monitoring_location_id``) while letting other APIs key off + This matches the historical Water Data behavior (sort by ``time``, + then ``monitoring_location_id``) while letting other APIs sort by their own columns (e.g. NGWMN's ``sample_time``). Parameters @@ -331,7 +327,7 @@ def _sort_rows(df: pd.DataFrame, dialect: OgcDialect) -> pd.DataFrame: # Matches a lowercase letter or digit immediately followed by an uppercase # letter — the camelCase/PascalCase word boundary where a ``_`` is inserted. -# A letter/digit boundary is intentionally NOT split (so ``navd88`` stays put). +# A letter/digit boundary is intentionally not split (so ``navd88`` is unchanged). _CAMEL_BOUNDARY_RE = re.compile(r"([a-z0-9])([A-Z])") @@ -372,7 +368,7 @@ def _finalize_ogc( ) -> tuple[pd.DataFrame, BaseMetadata]: """Shape a combined OGC result into the user-facing ``(df, md)``. - The single home for the OGC getters' result shaping: empties + The one place the OGC getters' results are shaped: empty results normalized, column names normalized to snake_case, types coerced (when ``convert_type``), the wire ``id`` renamed and columns ordered, rows sorted, optionally truncated to ``max_rows``, and the response wrapped @@ -384,7 +380,7 @@ def _finalize_ogc( ``max_rows`` is applied here (after dedup/sort, on the *combined* frame) rather than only per-chunk, so a chunked call's total is bounded - to exactly ``max_rows`` and a resumed call honors the cap too. The + to exactly ``max_rows`` and a resumed call applies the cap too. The per-page ``row_cap`` bound in the engine is only an early-stop download bound. ``base_url`` is required and captured with the finalizer so resumed calls query the same API's schema when their combined result is empty. @@ -399,11 +395,11 @@ def _finalize_ogc( geopd=geopd, include_geometry=include_geometry, ) - # Normalize to PEP-8 snake_case column names *first*, so the dialect's + # Normalize to PEP-8 snake_case column names first, so the dialect's # ``time_cols``/``numerical_cols``/``sort_cols`` (all snake_case) match # regardless of whether the API returns snake_case (Water Data, where # this is a no-op) or camelCase (a sibling OGC API). Doing it before - # type coercion is what makes ``convert_type`` reach a camelCase field. + # type coercion is what makes ``convert_type`` apply to a camelCase field. renames = { col: snake for col in frame.columns diff --git a/dataretrieval/progress.py b/dataretrieval/progress.py index 3d2acff82..c64a5e54d 100644 --- a/dataretrieval/progress.py +++ b/dataretrieval/progress.py @@ -1,22 +1,22 @@ """A single self-updating status line for paginated and chunked queries. -Retrieval adapters can fan out in ways the caller cannot see: large multi-value +Retrieval adapters can fan out without the caller being told: large multi-value requests are split into URL-length-safe *chunks* (``chunking`` module), and each request follows ``next`` links across an unknown number of *pages* -(``transport.pagination.paginate``). This module surfaces that work as one +(``transport.pagination.paginate``). This module reports that work as one line on stderr, rewritten in place as data arrives:: Retrieving: daily · 6 pages · 2,881 rows · 995/1,000 requests remaining -The active reporter lives in a :class:`~contextvars.ContextVar` rather than being +The active reporter is held in a :class:`~contextvars.ContextVar` rather than being threaded through every signature: progress is a cross-cutting concern that the chunk orchestrator (outer, chunk counts) and the page-walking loop (inner, -page/row/rate-limit counts) both update without knowing about each other. Call -:func:`progress_context` to activate one and :func:`current` to reach it. +page/row/rate-limit counts) both update without importing each other. Call +:func:`progress_context` to activate one and :func:`current` to get it. This is a top-level leaf rather than part of :mod:`dataretrieval.transport`: transport modules report *into* it and the execution layer owns no rendering -(ADR 0006), so every service adapter -- OGC or not -- reaches the same reporter. +(ADR 0006), so every service adapter -- OGC or not -- uses the same reporter. By default the line is shown for interactive use — an interactive terminal or a Jupyter/IPython kernel, like ``tqdm`` — while redirected logs and CI get no line. @@ -49,7 +49,7 @@ def _group_int(value: str) -> str: # The reporter active for the current query. A ContextVar (not a module global) # so the chunk orchestrator and the page loop resolve to the same reporter -# within one query, and an unrelated query in another context can't clobber its +# within one query, and an unrelated query in another context cannot overwrite its # state. (It does not give concurrent queries sharing one stderr separate # lines — they would still interleave.) _active: Ambient[ProgressReporter | None] = Ambient("dataretrieval_progress", None) @@ -61,12 +61,11 @@ def _group_int(value: str) -> str: def _in_jupyter_kernel() -> bool: """True when running inside a Jupyter/IPython *kernel* (notebook, lab, qtconsole). - A kernel's ``stderr`` isn't a TTY, but it honors carriage-return rewrites in - the cell output area — the same mechanism ``tqdm`` rides on — so the line is - worth showing there. The plain IPython terminal REPL is a - ``TerminalInteractiveShell`` (already a TTY), so only the ZMQ kernel needs - this extra signal. Detected without importing IPython: if it isn't already - imported, we aren't in a shell. + A kernel's ``stderr`` isn't a TTY, but it handles carriage-return rewrites in the + cell output area — the same mechanism ``tqdm`` uses — so the line is shown there. + The plain IPython terminal REPL is a ``TerminalInteractiveShell`` (already a TTY), + so only the ZMQ kernel needs this extra signal. Detected without importing IPython: + if it is not already imported, no IPython shell is running. """ ipython = sys.modules.get("IPython") if ipython is None: @@ -78,13 +77,13 @@ def _in_jupyter_kernel() -> bool: def _enabled_default(stream: TextIO) -> bool: """Whether to draw the line by default. - ``API_USGS_PROGRESS`` wins when set. Otherwise show it for interactive use — - a TTY or a Jupyter/IPython kernel — and stay quiet for redirected output, - logs, and CI. + ``API_USGS_PROGRESS`` takes precedence when set. Otherwise show it for + interactive use — a TTY or a Jupyter/IPython kernel — and draw nothing for + redirected output, logs, and CI. """ # config owns the grammar, so this is already a bool: the same value means # the same thing whether it came from a configure() block, the environment, - # or the file. Re-parsing here is what let those three disagree. + # or the file. Re-parsing here is what let those three differ. override = _configuration.progress() if override is not None: return override @@ -126,10 +125,10 @@ def __init__( # denominator when the server reports it. self.rate_limit: str | None = None # Transient note shown while a chunk backs off before a - # retry; cleared by the next page/chunk so it doesn't linger. + # retry; cleared by the next page/chunk so it is not left on the line. self.retry_note: str | None = None self._last_len = 0 - # Whether anything was actually written to the stream — drives whether + # Whether anything was written to the stream — decides whether # close() needs a terminating newline. (``current_chunk`` doesn't # track that: ``start_chunk`` sets it even when it doesn't render.) self._rendered = False @@ -142,7 +141,7 @@ def set_chunks(self, total: int) -> None: def start_chunk(self, index: int) -> None: """Mark the start of chunk ``index`` (1-based) and redraw. - Only redraws when actually chunking (``total_chunks > 1``); a + Only redraws when chunking (``total_chunks > 1``); a single-chunk plan has nothing chunk-specific to show yet, so it avoids a premature "0 pages" frame before the first page arrives. """ @@ -152,7 +151,7 @@ def start_chunk(self, index: int) -> None: self._render() def add_page(self, rows: int = 0) -> None: - """Record one fetched page carrying ``rows`` rows and redraw.""" + """Record one fetched page of ``rows`` rows and redraw.""" self.pages += 1 self.rows += int(rows) self.retry_note = None @@ -165,7 +164,7 @@ def note_retry(self, *, attempt: int, wait: float) -> None: :meth:`close`) so the line returns to normal once the retry resolves. """ # Keep sub-second waits explicit (avoid misleading ``0s``) while - # rendering whole-second waits without unnecessary ``.0`` noise. + # rendering whole-second waits without an unnecessary ``.0``. # ``float()`` to support Python 3.9-3.11: ``round(int, 1)`` returns an # int and ``int.is_integer()`` (used below) only exists on 3.12+. wait_1dp = round(float(wait), 1) @@ -182,9 +181,8 @@ def set_rate_remaining( """Update the rate-limit display from the response headers. ``value`` is ``x-ratelimit-remaining``; ``limit`` is the optional - ``x-ratelimit-limit`` quota, shown as the denominator. Empty/missing - values are ignored so a page that omits a header doesn't blank out the - last known value. + ``x-ratelimit-limit`` quota, shown as the denominator. Empty/missing values are + ignored so a page that omits a header does not clear the last known value. """ if value not in (None, ""): self.rate_remaining = str(value) @@ -223,10 +221,10 @@ def _render(self) -> None: self._last_len = len(line) self._rendered = True except Exception: # noqa: BLE001 - # Progress output is best-effort cosmetics; a broken pipe (output + # Progress output is best-effort display; a broken pipe (output # piped to ``head``), a closed stream, or an encoding error must - # never disturb — let alone truncate — the query. Disable so we - # don't retry on every subsequent page. + # never affect the query. Disable so rendering + # is not retried on every subsequent page. self.enabled = False def close(self) -> None: @@ -234,11 +232,11 @@ def close(self) -> None: If the query targeted the API-key host and no key is configured (no ``API_USGS_PAT``), append a one-time pointer to API-key registration, - since unauthenticated callers hit much lower rate limits. + since unauthenticated callers are subject to much lower rate limits. """ if self._closed: return - # A retry note set during the final backoff would otherwise freeze as + # A retry note set during the final backoff would otherwise remain as # the persisted last line of a call that has since completed or given # up; clear it and redraw (while still un-closed, so ``_render`` runs) # so the final state isn't a stale "retrying". @@ -260,7 +258,7 @@ def _maybe_hint_api_key(self) -> None: if not self._key_helps or _api_key_hint_shown or api_key(): return # Set the once-per-process latch only after a successful write, so a - # failed write (broken pipe) doesn't silently burn the hint for every + # failed write (broken pipe) doesn't suppress the hint for every # later query in the process. self._stream.write( f"No API key detected — register for higher rate limits at {SIGNUP_URL}\n" @@ -279,8 +277,8 @@ def progress_context( """Activate a :class:`ProgressReporter` for the duration of a query. ``service`` labels the line (e.g. ``"Retrieving: daily ..."``), and - ``target_url`` is where the query is going -- it decides whether an - API-key pointer is worth showing when the line closes. If a reporter is + ``target_url`` is the host the query is sent to -- it decides whether an + API-key pointer is shown when the line closes. If a reporter is already active (a nested call), the existing one is yielded unchanged so the outermost query owns the single line; only the outermost context closes it (and every argument of a nested call is ignored). diff --git a/dataretrieval/rdb.py b/dataretrieval/rdb.py index 989b0aae7..4aa33df7c 100644 --- a/dataretrieval/rdb.py +++ b/dataretrieval/rdb.py @@ -4,7 +4,7 @@ and by the Water Data STAC catalog's rating-curve assets. Every RDB file has the same shape: -- One or more ``#``-prefixed comment lines carrying provenance metadata +- One or more ``#``-prefixed comment lines holding provenance metadata (data source, retrieval timestamp, station name, parameter codes, etc.). - A tab-separated header row naming each column. - A second tab-separated row giving column format specs (e.g. ``5s 15s``); @@ -33,7 +33,7 @@ def read_rdb(text: str, dtypes: dict[str, type] | None = None) -> pd.DataFrame: The RDB text response from a USGS web service. dtypes : dict[str, type] or None, optional Column-name to dtype hints, forwarded to ``pandas.read_csv``. Unknown - column names are silently ignored, so callers can pass a dict of every + column names are ignored, so callers can pass a dict of every column they might be interested in. Returns @@ -85,10 +85,9 @@ def read_rdb(text: str, dtypes: dict[str, type] | None = None) -> pd.DataFrame: def extract_rdb_comment(text: str) -> list[str]: """Return the RDB ``#``-prefixed comment block, raw and in original order. - Each entry includes its leading ``#`` and any whitespace, matching what - R's ``dataRetrieval`` returns from ``comment(df)``. The comment block - carries provenance metadata that is otherwise lost during parsing — - data source, retrieval timestamp, parameter codes, rating id and - last-shifted timestamp for ratings, etc. + Each entry includes its leading ``#`` and any whitespace, matching what R's + ``dataRetrieval`` returns from ``comment(df)``. The comment block holds provenance + metadata that is otherwise lost during parsing — data source, retrieval timestamp, + parameter codes, rating id and last-shifted timestamp for ratings, etc. """ return [line for line in text.splitlines() if line.startswith("#")] diff --git a/dataretrieval/streamstats.py b/dataretrieval/streamstats.py index 91a18ffa9..f8c14e2c4 100644 --- a/dataretrieval/streamstats.py +++ b/dataretrieval/streamstats.py @@ -36,8 +36,8 @@ def _service_base() -> str: """The StreamStats base this call targets: a block's redirect, or its own. - Both endpoints below hang off this, so a - ``StreamstatsConfiguration(base_url=...)`` moves the whole service rather + Both endpoints below are built on this, so a + ``StreamstatsConfiguration(base_url=...)`` redirects the whole service rather than one endpoint (ADR 0011). """ return _configuration.base_url(adapter="streamstats", default=STREAMSTATS_URL) @@ -174,8 +174,8 @@ def get_watershed( return r if format == "shape": - # Returning a shapefile/Fiona object isn't implemented; fail - # loudly instead of silently falling through to a Watershed. + # Returning a shapefile/Fiona object isn't implemented; raise + # instead of falling through to a Watershed. raise NotImplementedError( "format='shape' is not implemented. Use format='geojson' " "(default) for the raw response, or format='object' for a " @@ -224,7 +224,7 @@ def __init__(self, rcode: str, xlocation: float, ylocation: float) -> None: def from_streamstats_json(cls, streamstats_json: dict[str, Any]) -> Watershed: """Create a :class:`Watershed` from a parsed StreamStats JSON payload. - No new request is issued. Builds a fresh instance (via ``__new__``, so + No new request is issued. Builds a new instance (via ``__new__``, so the network-fetching ``__init__`` is bypassed) and populates it; each call returns an independent object rather than mutating shared class state. @@ -245,7 +245,7 @@ def _populate(self, streamstats_json: dict[str, Any]) -> None: class StreamstatsConfiguration(_Redirectable, _Retrying, BaseConfiguration): """Settings for StreamStats calls alone. - No fan-out dials: a StreamStats query is answered by a single + No fan-out settings: a StreamStats query is served by a single request. Declared here rather than in :mod:`dataretrieval.configuration` @@ -260,7 +260,7 @@ class StreamstatsConfiguration(_Redirectable, _Retrying, BaseConfiguration): stops. base_url : str, optional Services base to send StreamStats requests to, instead of its own - (``STREAMSTATS_URL``). Both endpoints hang off it. Code only: + (``STREAMSTATS_URL``). Both endpoints are built on it. Code only: the file and the environment refuse it. """ diff --git a/dataretrieval/transport/fanout.py b/dataretrieval/transport/fanout.py index 84a5a87b9..e135d92e8 100644 --- a/dataretrieval/transport/fanout.py +++ b/dataretrieval/transport/fanout.py @@ -5,44 +5,42 @@ multi-value axes by :class:`dataretrieval.ogc.planning.ChunkPlan`; or a Water Use query naming several locations, which the NWDC accepts only one at a time. -This module owns distribution and nothing else: concurrency bounded by a -semaphore, per-attempt retry, deterministic failure precedence, sparse -completion tracking, and resume. It names no protocol concept -- an adapter -supplies a :class:`FanOutPlan` (whatever structure it divided into, if any) and -an ``async def fetch(item) -> (df, response)``. Dividing a query is protocol -knowledge and stays in OGC; distributing the pieces is protocol-neutral and -lives here. That split is ADR 0008. +This module is responsible only for distribution: concurrency bounded by a semaphore, +per-attempt retry, deterministic failure precedence, sparse completion tracking, and +resume. It names no protocol concept -- an adapter supplies a :class:`FanOutPlan` +(whatever structure it divided into, if any) and an ``async def fetch(item) -> (df, +response)``. Dividing a query is protocol-specific and is defined in OGC; distributing +the pieces is protocol-neutral and is defined here. That split is ADR 0008. Concurrency: :meth:`FanOut._run` dispatches every pending chunk under one ``asyncio.gather`` sharing a single ``httpx.AsyncClient``. An -``asyncio.Semaphore`` -- not the client's connection pool, which is merely sized +``asyncio.Semaphore`` -- not the client's connection pool, which is only sized to match -- caps the chunks in flight at ``N`` (ADR 0008). The ``concurrency`` setting resolves ``N`` -- a ``configure()`` block, then ``API_USGS_CONCURRENT``, then the config file, and per adapter as well as package-wide: an integer N > 1 allows N chunks in flight; ``1`` forces -sequential dispatch; the literal ``unbounded`` lifts the cap. ``N`` bounds only +sequential dispatch; the literal ``unbounded`` removes the cap. ``N`` bounds only how many of a query's chunks are in flight at once -- a client-side trade-off between open connections and fan-out latency. It does not affect the API rate limit: a fanned-out call issues the same number of chunks regardless of ``N``, so ``N`` changes their timing, not the total request volume. The USGS API rate-limits by volume over time (HTTP 429), not by simultaneity; set ``API_USGS_PAT`` to raise that quota. The default of 32 is a -conservative cap that keeps connection use modest. The fan-out runs in a +conservative cap that keeps connection use low. The fan-out runs in a short-lived worker thread (an ``anyio`` blocking portal), so it works whether or not the caller is already inside an event loop (Jupyter / IPython / async apps). -Retries: each chunk is retried on a transient failure (429, 5xx, -connect/read timeout) with exponential backoff + full jitter, honoring a server -``Retry-After`` when present. The ``retries`` setting caps them (default 4; -``0`` disables), resolved through the same chain and scopable per adapter. A -``Retry-After`` longer than the per-call ceiling escalates to a resumable -interruption. - -Interruption: any mid-stream transient failure surfaces as a -:class:`~dataretrieval.interruptions.FanOutInterrupted` subclass carrying -``.call``, a :class:`FanOut` handle owning the already-completed chunk -state. Call ``.call.resume()`` once the underlying condition clears; only the -still-pending chunks are re-issued. +Retries: each chunk is retried on a transient failure (429, 5xx, connect/read timeout) +with exponential backoff + full jitter, applying a server ``Retry-After`` when present. +The ``retries`` setting caps them (default 4; ``0`` disables), resolved through the same +chain and scopable per adapter. A ``Retry-After`` longer than the per-call ceiling is +raised as a resumable interruption. + +Interruption: any transient failure partway is raised as a +:class:`~dataretrieval.interruptions.FanOutInterrupted` subclass with ``.call``, a +:class:`FanOut` handle holding the already-completed chunk state. Call +``.call.resume()`` once the underlying condition has ended; only the still-pending +chunks are re-issued. """ from __future__ import annotations @@ -72,19 +70,19 @@ from dataretrieval.transport.retry import _NO_RETRY, RetryPolicy from dataretrieval.transport.retry import retry_async as _retry -#: One chunk's description, as the adapter's ``fetch`` wants it. The +#: One chunk's description, in the form the adapter's ``fetch`` accepts. The #: executor never inspects it — see :class:`FanOutPlan`. _Chunk = TypeVar("_Chunk") -#: The same thing in :class:`FanOutPlan`, where it only ever comes *out* of the -#: plan. Covariant so a ``list[httpx.Request]`` satisfies a plan of any -#: supertype, the way ``Iterable`` is covariant for the same reason. +#: The same thing in :class:`FanOutPlan`, where it is only ever produced by the plan. +#: Covariant so a ``list[httpx.Request]`` satisfies a plan of any supertype, the way +#: ``Iterable`` is covariant for the same reason. _ChunkCo = TypeVar("_ChunkCo", covariant=True) # The fan-out concurrency cap resolves through # :func:`dataretrieval.configuration.concurrency`, which owns the setting's name, its -# grammar (``1`` sequential, >1 bounded, ``unbounded`` uncapped) and its -# built-in default. Naming any of those here too would let this module and the -# chain disagree about what a value means. +# grammar (``1`` sequential, >1 bounded, ``unbounded`` uncapped) and its built-in +# default. Naming any of those here too would let this module and the chain define a +# value differently. # --------------------------------------------------------------------------- @@ -96,12 +94,12 @@ class FanOutPlan(Protocol[_ChunkCo]): """ The contract a plan satisfies for a fan-out to execute it. - A **plan** is defined in ``CONTEXT.md``. This protocol is that enumeration - and nothing more: ``len`` must agree with the number of items iteration - yields, and iteration must repeat the same items in the same order, since + A **plan** is defined in ``CONTEXT.md``. This protocol is that enumeration and + nothing more: ``len`` must equal the number of items iteration yields, and iteration + must repeat the same items in the same order, since :meth:`FanOut.resume` keys completed work by position. The item type is whatever an adapter's own ``fetch`` accepts -- this executor passes each - item through untouched and never inspects it, so the OGC getters yield + item through unchanged and never inspects it, so the OGC getters yield kwargs dicts while Water Use yields ready :class:`httpx.Request` objects. The query's ``canonical_url`` is not part of the plan; it is an argument to :class:`FanOut`. @@ -120,7 +118,7 @@ def __iter__(self) -> Iterator[_ChunkCo]: ... # Shared per-call client # --------------------------------------------------------------------------- -# The per-call ``httpx.AsyncClient``, published for the duration of +# The per-call ``httpx.AsyncClient``, set for the duration of # ``FanOut._run`` so paginated-loop helpers reuse the same connection pool # across every chunk. ``None`` outside a fan-out — paginated helpers then # open their own short-lived client. Deliberately a plain ContextVar-backed @@ -131,14 +129,14 @@ def __iter__(self) -> Iterator[_ChunkCo]: ... def active_client() -> httpx.AsyncClient | None: """ - Return the fan-out's currently-published client, or ``None``. + Return the fan-out's currently set client, or ``None``. Used by paginated-loop helpers to reuse the per-call connection pool. Returns ------- httpx.AsyncClient or None - The client published for the duration of a :meth:`FanOut._run`; + The client set for the duration of a :meth:`FanOut._run`; ``None`` outside one. """ return _active_client.get() @@ -148,7 +146,7 @@ def active_client() -> httpx.AsyncClient | None: # Type aliases for the FanOut contract # --------------------------------------------------------------------------- -# The per-chunk fetcher an adapter injects and ``FanOut`` drives: an +# The per-chunk fetcher an adapter injects and ``FanOut`` calls: an # ``async def fetch(item) -> (df, response)``, where ``item`` is whatever the # adapter's plan yields. _Fetch = Callable[[_Chunk], Awaitable[tuple[pd.DataFrame, httpx.Response]]] @@ -173,28 +171,28 @@ class FanOut(Generic[_Chunk]): Stateful handle for a fanned-out call. Holds the in-flight state (per-chunk frames and responses) - and the async fetcher. A single :meth:`resume` entry point drives - the call from wherever it is to completion — used both for the + and the async fetcher. A single :meth:`resume` entry point runs + the call from its current state to completion — used both for the first invocation and for subsequent retries after a :class:`~dataretrieval.interruptions.FanOutInterrupted`. :meth:`_run` gathers every pending chunk over one shared :class:`httpx.AsyncClient`, applies the failure-precedence rules, and - combines; :meth:`resume` drives it through an ``anyio`` blocking + combines; :meth:`resume` runs it through an ``anyio`` blocking portal so it works whether or not the caller is already inside an event loop. Concurrency is bounded by a per-run ``asyncio.Semaphore`` (see :meth:`_run`), so sequential dispatch - (``API_USGS_CONCURRENT=1``) is just a degenerate gather. + (``API_USGS_CONCURRENT=1``) is a gather with a semaphore of one. A ``FanOut`` is created internally when an adapter executes a plan; - callers reach it via ``FanOutInterrupted.call`` on the exception raised - by a mid-stream failure. + callers obtain it from ``FanOutInterrupted.call`` on the exception raised + by a failure partway through. :meth:`resume` is idempotent: :meth:`_run` iterates the plan (deterministic order) and skips any index whose result is already in ``self._chunks``. The completion set is a sparse ``dict[int, (df, response)]`` so the - gather can record scattered completions (e.g. indices [0, 2, 5] + gather can record non-contiguous completions (e.g. indices [0, 2, 5] after siblings [1, 3, 4] failed) and a subsequent ``resume`` only re-issues the missing indices. @@ -216,24 +214,24 @@ class FanOut(Generic[_Chunk]): canonical_url : str or None, optional URL identifying the query as a whole, restored onto the combined response so the caller sees the request they made rather than - whichever chunk happened to land last. Also the destination + whichever chunk completed last. Also the destination :meth:`resume` labels its progress line with. service : str or None, optional - Human-facing name of what is being retrieved (e.g. ``"daily"``, + Display name of what is being retrieved (e.g. ``"daily"``, ``"nwdc"``), used to label the progress line :meth:`resume` opens. ``None`` leaves the line unlabelled. Attributes ---------- plan : FanOutPlan - The plan being driven (read-only after construction). + The plan being run (read-only after construction). fetch : Callable The async per-chunk fetch function. finalize : Callable - Transform applied to the combined result (see :data:`_Finalize`) at - the terminal :meth:`_run` return, so a completed call yields the - caller's finished shape. The ``partial_*`` accessors deliberately - skip it and stay raw. + Transform applied to the combined result (see :data:`_Finalize`) at the terminal + :meth:`_run` return, so a completed call yields the caller's finished shape. The + ``partial_*`` accessors deliberately skip it and return the raw frame and + response. partial_frame : pandas.DataFrame Raw combined frame of completed chunks (live; recomputed per access). Not finalized — call :meth:`resume` for the finished shape. @@ -260,35 +258,35 @@ def __init__( self.retry_policy = retry_policy self.finalize = finalize self.canonical_url = canonical_url - # Label for the progress line :meth:`resume` opens. It lives here, next + # Label for the progress line :meth:`resume` opens. It is here, next # to ``canonical_url``, because this class is what emits the progress # events — see :meth:`resume`. self.service = service - # Which adapter's settings this drive resolves, so a ``[ngwmn]`` table - # or an ``NgwmnConfiguration`` reaches only NGWMN calls. Distinct from + # Which adapter's settings this run resolves, so a ``[ngwmn]`` table + # or an ``NgwmnConfiguration`` applies only to NGWMN calls. Distinct from # ``service`` above, which is a *display label* for the progress line # and is variously a collection name or prose. ``None`` resolves # package-wide. See ADR 0010. self.adapter = adapter - # This service's preferred cap for when nothing is configured. Resolved - # at resume time, not here, so a setting that arrives after this call - # was built still applies. Anything the chain resolves outranks it -- - # see :func:`dataretrieval.configuration.concurrency` for why a service - # preference must not override an explicit setting. + # This service's preferred cap for when nothing is configured. Resolved at + # resume time, not here, so a setting made after this call was built still + # applies. Anything the chain resolves outranks it -- see + # :func:`dataretrieval.configuration.concurrency` for why a service preference + # must not override an explicit setting. self.default_concurrent = default_concurrent # Extra ``httpx.AsyncClient`` options merged into the shared client this # run opens (``verify`` for the Water Use ``ssl_check`` flag, say). The # executor owns client lifecycle, so an adapter with a per-call client - # requirement has to hand it down rather than open its own — opening its - # own would defeat the shared connection pool. Empty for OGC, which + # requirement has to pass it in rather than open its own — opening its + # own would bypass the shared connection pool. Empty for OGC, which # exposes no such flag. self.client_options = client_options or {} # No ambient state is snapshotted here: everything a chunk rebuild # needs (base URL, dialect, row cap for the OGC getters) is closed # over by the adapter's ``fetch``/plan, so a *later* - # ``exc.call.resume()`` — fired after the originating call + # ``exc.call.resume()`` — invoked after the originating call # returned — rebuilds chunks against the values the call was - # created with without this executor carrying adapter state. + # created with without this executor holding adapter state. # Completed (frame, response) pairs keyed by sub-args index; sparse # (gathered chunks complete out of order — see class docstring). # ``_run``'s ``track`` closure is the only writer, so ``dict`` insertion @@ -297,11 +295,10 @@ def __init__( def wrap_failure(self, exc: BaseException) -> FanOutInterrupted | None: """ - Build the matching :class:`FanOutInterrupted` carrying this - call when ``exc`` is a recognized transient transport failure; - return ``None`` for unrecognized failures so the caller can - re-raise. Encapsulates the - ``classify → instantiate-with-call-state`` recipe so + Build the matching :class:`FanOutInterrupted` that holds this call when ``exc`` + is a recognized transient transport failure; return ``None`` for unrecognized + failures so the caller can re-raise. Encapsulates the + classify-then-instantiate-with-call-state sequence so :class:`FanOut`'s private fields stay private. Parameters @@ -312,8 +309,8 @@ def wrap_failure(self, exc: BaseException) -> FanOutInterrupted | None: Returns ------- FanOutInterrupted or None - The matching :class:`FanOutInterrupted` subclass carrying this - call for a recognized transient failure; ``None`` otherwise. + The matching :class:`FanOutInterrupted` subclass holding this call for a + recognized transient failure; ``None`` otherwise. """ classification = _classify_chunk_error(exc) if classification is None: @@ -379,8 +376,9 @@ def partial_frame(self) -> pd.DataFrame: Live — recomputed on each access so it reflects current state across resume attempts. Deliberately the *raw* combined frame - (``_combine_frames``), NOT the finalized result: this is a cheap, - side-effect-free snapshot for inspecting partial progress, so + (``_combine_frames``), not the finalized result: this is a + side-effect-free snapshot for inspecting partial progress, with no + network I/O, so reading it (or building a :class:`FanOutInterrupted` around it) never triggers ``finalize`` work — which for OGC getters includes a schema network fetch on an empty frame. Use ``call.resume()`` @@ -417,9 +415,9 @@ def _pending(self) -> Iterator[tuple[int, _Chunk]]: Yield ``(index, item)`` for chunks not yet completed. Iterates the plan in its deterministic order and skips any index - already in ``self._chunks``. :meth:`_run` uses this to pick up - exactly the chunks it still owes — the mechanism behind - idempotent resume. + already in ``self._chunks``. :meth:`_run` uses this to run only + the chunks not yet completed, which is what makes resume + idempotent. """ for index, item in enumerate(self.plan): if index not in self._chunks: @@ -427,25 +425,25 @@ def _pending(self) -> Iterator[tuple[int, _Chunk]]: def resume(self) -> tuple[pd.DataFrame, Any]: """ - Drive the call to completion and return the combined result. + Run the call to completion and return the combined result. - Opens the progress line for the drive and runs :meth:`_run` through + Opens the progress line for the run and runs :meth:`_run` through an ``anyio`` blocking portal (a short-lived worker thread), so it works whether or not the caller is already inside an event loop (Jupyter / IPython / async apps). The portal copies the calling - context, so the active progress reporter still reaches the - chunks. + context, so the active progress reporter is still visible to + the chunks. This executor is what emits progress events, so it is also what owns - the reporter's lifetime: an adapter that drives a ``FanOut`` gets the - line for free instead of having to remember a separate + the reporter's lifetime: an adapter that runs a ``FanOut`` gets the + line without a separate ``with progress_context(...)`` block. A reporter already active (a nested getter, or a caller's own context) is reused unchanged. Idempotent: only chunks whose index isn't already in ``self._chunks`` are re-issued. Item order is the plan's own and is deterministic, so a partial completion (sparse indices) - resumes onto the same items. + resumes with the same items. Returns ------- @@ -461,31 +459,30 @@ def resume(self) -> tuple[pd.DataFrame, Any]: Raises ------ FanOutInterrupted - On a mid-stream transient failure — 429, 5xx, or a bare - transport error: :class:`~dataretrieval.interruptions.QuotaExhausted` - for 429, :class:`~dataretrieval.interruptions.ServiceInterrupted` - for the rest. The resumable handle is on ``exc.call`` — wait for - the underlying condition to clear and call ``exc.call.resume()`` - again. + On a transient failure partway through — 429, 5xx, or a bare transport + error: :class:`~dataretrieval.interruptions.QuotaExhausted` for 429, + :class:`~dataretrieval.interruptions.ServiceInterrupted` for the rest. The + resumable handle is on ``exc.call`` — wait for the underlying condition to + end and call ``exc.call.resume()`` again. """ # Open the line here, in the *calling* context, so an outer # reporter (a nested getter, or the caller's own ``progress_context``) - # is the one found and reused; a drive that finds none gets a fresh + # is the one found and reused; a run that finds none gets a new # line, which is what makes a resume long after the interruption - # report progress at all. ``start_blocking_portal`` copies this + # report progress. ``start_blocking_portal`` copies this # calling context into its worker thread, so the active reporter - # reaches the chunks. Chunk-rebuild state (base URL, dialect, row - # cap) travels in the adapter's ``fetch`` closure, not in ambient + # is visible to the chunks. Chunk-rebuild state (base URL, dialect, row + # cap) is held in the adapter's ``fetch`` closure, not in ambient # state, so no construction-time snapshot is needed for a resume - # fired after the originating call returned (see ``__init__``). + # invoked after the originating call returned (see ``__init__``). with _progress.progress_context( service=self.service, target_url=self.canonical_url ): - # Resolve concurrency here, per drive, rather than at construction. - # It is the one dial a caller adjusts precisely *while* retrying -- + # Resolve concurrency here, per run, rather than at construction. + # It is the one setting a caller changes *while* retrying -- # the documented recovery from QuotaExhausted is to wait and - # re-issue more gently -- so a ``configure()`` block entered - # between the interruption and the resume has to win. + # re-issue at lower concurrency -- so a ``configure()`` block entered + # between the interruption and the resume must take precedence. concurrency = _configuration.concurrency( self.default_concurrent, adapter=self.adapter ) @@ -506,18 +503,18 @@ def _handle_gather_failures( 1. Cancellation / interrupt signals (``CancelledError``, ``KeyboardInterrupt``, ``SystemExit`` — non-``Exception``) - propagate unmodified; wrapping them as a transient would swallow + propagate unmodified; wrapping them as a transient would suppress the user's stop signal. - 2. A non-transient failure (a real bug — unrecognized by - ``wrap_failure``) surfaces raw, so it isn't masked behind a - resumable handle for a transient sibling that landed later. - 3. Only when every failure is a recognized transient do we raise - the first as a resumable ``FanOutInterrupted``. - - ``wrap_failure`` is asked only for the one failure that is raised. - Asking it per failure would snapshot the combined frame N times (a + 2. A non-transient failure (a programming error — one + ``wrap_failure`` does not recognize) is raised unwrapped, so it is not masked + by a resumable handle for a transient sibling that completed later. + 3. Only when every failure is a recognized transient is the first raised as a + resumable ``FanOutInterrupted``. + + ``wrap_failure`` is called only for the one failure that is raised. + Calling it per failure would snapshot the combined frame N times (a full concat over every completed chunk) and discard all but one, - which a batch of chunks failing together makes routine. + which happens whenever a batch of chunks fails together. Raises ------ @@ -538,7 +535,7 @@ def _handle_gather_failures( first_transient = failures[0] interrupted = self.wrap_failure(first_transient) if interrupted is None: - # Unreachable: classified as transient just above. + # Unreachable: classified as transient above. raise self._normalize_failure(first_transient) raise interrupted from first_transient @@ -549,33 +546,32 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: Pending chunks (:meth:`_pending`) fan out under ``asyncio.gather`` with ``return_exceptions=True`` so completed - chunks survive a sibling's transient failure. On a + chunks are kept after a sibling's transient failure. On a recognized transient (:class:`~dataretrieval.exceptions.RateLimited`, :class:`~dataretrieval.exceptions.ServiceUnavailable`, or a bare - ``httpx.HTTPError`` / ``httpx.InvalidURL``) a - :class:`FanOutInterrupted` subclass is raised carrying ``self`` on - ``.call``; ``exc.call.resume()`` then re-issues only the unfinished - indices through this same runner. + ``httpx.HTTPError`` / ``httpx.InvalidURL``) a :class:`FanOutInterrupted` + subclass is raised with ``self`` on ``.call``; ``exc.call.resume()`` then + re-issues only the unfinished indices through this same runner. - The gather dispatches *every* pending chunk at once, but an + The gather dispatches every pending chunk at once, but an ``asyncio.Semaphore`` caps the number of concurrent fetches at - ``N = max_concurrent`` — ``None`` lifts the cap, ``N=1`` runs them + ``N = max_concurrent`` — ``None`` removes the cap, ``N=1`` runs them one at a time. The connection pool is sized to the same ``N`` (``httpx.Limits(max_connections=N, max_keepalive_connections=N)``) so the in-flight fetches reuse keepalive connections. The semaphore, not the pool, is the throttle (ADR 0008); holding chunks at the semaphore keeps them out of the pool, so the pool - timeout only fires for a genuinely stuck connection. + timeout expires only for a connection that is stuck. - The shared client is published on :data:`_active_client` so + The shared client is set on :data:`_active_client` so the paginated-loop helpers reuse its connection pool. Parameters ---------- max_concurrent : int or None Maximum chunks in flight (the semaphore value, and the - connection-pool size). ``None`` lifts the cap entirely. + connection-pool size). ``None`` removes the cap. Returns ------- @@ -595,10 +591,10 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: re-issues the unfinished ones. """ # At httpx's default client limits (``max_connections=100``, - # keepalive 20), the pool would bottleneck a wider cap or churn - # connections by keeping too few alive. ``unbounded`` - # (``max_concurrent=None``) is a degenerate cap at the plan total — a - # semaphore that can never block — so gated is the only code path. + # keepalive 20), the pool would limit a wider cap, or would open and close + # connections repeatedly by keeping too few alive. ``unbounded`` + # (``max_concurrent=None``) is a cap equal to the plan total — a + # semaphore that can never block — so the gated path is the only code path. limits = httpx.Limits( max_connections=max_concurrent, max_keepalive_connections=max_concurrent ) @@ -615,14 +611,14 @@ async def _run(self, max_concurrent: int | None) -> tuple[pd.DataFrame, Any]: async def track( index: int, item: _Chunk ) -> tuple[pd.DataFrame, httpx.Response]: - """One chunk (with retry) + result-store + progress tick.""" + """One chunk (with retry) + result-store + progress update.""" result = await _retry( lambda: self.fetch(item), self.retry_policy, gate=semaphore ) self._chunks[index] = result if reporter is not None: - # Chunks finish out of order under gather, so tick the - # completed *count* rather than a positional index. + # Chunks finish out of order under gather, so report the + # completed count rather than a positional index. reporter.start_chunk(self.completed_chunks) return result diff --git a/dataretrieval/transport/http.py b/dataretrieval/transport/http.py index b14d5bb61..82029f901 100644 --- a/dataretrieval/transport/http.py +++ b/dataretrieval/transport/http.py @@ -18,9 +18,9 @@ ) from dataretrieval.exceptions import NetworkError -# Re-exported for the adapters that reach for credential policy through the -# transport surface they already import. ``dataretrieval.credentials`` is the -# single definition; these names are views on it, not copies of it. +# Re-exported for the adapters that import credential policy through the +# transport module they already import. ``dataretrieval.credentials`` is the +# single definition; these names refer to the same objects; they are not copies. __all__ = [ "HTTPX_ASYNC_DEFAULTS", "HTTPX_DEFAULTS", @@ -50,12 +50,12 @@ def default_headers(target_url: str | httpx.URL | None = None) -> dict[str, str]: """Build standard headers, scoping the API key to its authorized host. - The host is checked *before* the key is resolved, and the key is resolved - only for the authorized host. Order matters now that settings come from a - layered chain: resolution reads the config file and can raise + The host is checked *before* the key is resolved, and the key is resolved only for + the authorized host. Order matters because settings come from a layered chain: + resolution reads the config file and can raise :class:`~dataretrieval.exceptions.ConfigurationError` for a malformed file or a profile it no longer defines. Resolving first would let a Water Data - configuration problem break a legacy NWIS, WQP, or NGWMN call that would + configuration problem fail a legacy NWIS, WQP, or NGWMN call that would never have received the key. """ headers = { diff --git a/dataretrieval/transport/links.py b/dataretrieval/transport/links.py index c438f5492..936b161e9 100644 --- a/dataretrieval/transport/links.py +++ b/dataretrieval/transport/links.py @@ -1,19 +1,19 @@ """One policy for the server-supplied next-page links every page walk follows. -A ``next`` href is response *data*, not configuration: it arrives over the wire -from the service (or from whatever answered for it) and then becomes the URL of -our next request, carrying that request's headers and API key. Three page walks +A ``next`` href is response *data*, not configuration: it arrives in the response +from the service (or from whatever responded in its place) and then becomes the URL of +the next request, with that request's headers and API key. Three page walks -- the OGC engine's ``links`` array, the ratings STAC walk, and Water Use's -``Link:`` header -- each need the same things of it before it is trusted: parse -it, resolve a relative reference against the page it came from, refuse a host -the caller never asked for, and drop any embedded ``user:pass@`` (which +``Link:`` header -- each need the same things of it before it is used: parse +it, resolve a relative reference against the page it came from, reject a host +the caller did not request, and drop any embedded ``user:pass@`` (which ``httpx`` would otherwise turn into an ``Authorization: Basic`` header). -They had three implementations of that policy, and the three disagreed: only two -resolved relative references, only two refused an unparseable link rather than -handing it back, and each worded its refusal differently. A security invariant -with three spellings is one that gets fixed in one place and stays broken in the -other two, so it lives here once and the walks pass in what genuinely differs -- +They had three implementations of that policy, and the three differed: only two +resolved relative references, only two rejected an unparseable link rather than +returning it, and each worded its refusal differently. Three copies of a +security check diverge when one is fixed and the others are not, so it is +defined here once and the callers pass in what differs -- which hosts are acceptable, and whether an accepted host is rewritten. """ @@ -30,7 +30,7 @@ def _page_url(response: httpx.Response) -> httpx.URL: """The URL *response* came from, as an ``httpx.URL``. - Read lazily by :func:`resolve_next_url` (see there). The coercion is for + Read lazily by :func:`resolve_next_url` (see that function). The coercion is for callers holding a response-shaped stand-in whose ``url`` is a plain string. """ url = response.url @@ -48,37 +48,37 @@ def resolve_next_url( ) -> str: """Return *href* as a URL safe to request, or raise if it is not. - ``response.url`` is consulted only when it is actually needed -- to resolve a - relative reference, or as the default acceptable host. A walk that names its - own acceptable hosts and receives an absolute link never touches it, which - keeps this usable on a response whose request was never attached. + ``response.url`` is read only when it is needed -- to resolve a relative reference, + or as the default acceptable host. A caller that names its own acceptable hosts and + receives an absolute link never reads it, which keeps this usable on a response + whose request was never attached. Parameters ---------- href : str The next-page link exactly as the service supplied it. response : httpx.Response - The page the link arrived on. + The page that contained the link. service : str Name of the service, used in the error messages (e.g. ``"ratings"``). allowed_hosts : frozenset of str, optional - Hosts the link may name. Defaults to just the responding host; pass a - wider set only where the service is known to spell its own host several + Hosts the link may name. Defaults to the responding host alone; pass a + wider set only where the service is known to name its own host several ways. rewrite_host : str, optional Rewrite an accepted link to this host over ``https``, dropping any - explicit port. For a service whose links name a spelling of the host + explicit port. For a service whose links name a hostname that does not serve the API. error : type of Exception, optional Exception type to raise. Defaults to :class:`~dataretrieval.exceptions.DataRetrievalError`; the OGC engine - passes ``RuntimeError`` to keep the type it has always raised, until - retyping it is a deliberate, released decision. + passes ``RuntimeError`` to keep the type it has always raised, since changing + that type would be a released behavior change. Returns ------- str - An absolute URL on an acceptable host, carrying no embedded credentials. + An absolute URL on an acceptable host, with no embedded credentials. """ try: target = httpx.URL(href) @@ -97,22 +97,22 @@ def resolve_next_url( ) if target.host not in expected: raise error( - f"Refusing to follow a cross-host next-page link: the {service} " + f"Not following a cross-host next-page link: the {service} " f"response points at {target.host} rather than " f"{rewrite_host or ' or '.join(sorted(expected))}. Following it " f"would send this request, and any credentials on it, to a host " - f"you did not ask for. Retrying will not help; report this if it " + f"was not requested. Retrying will not help; report this if it " f"persists." ) if rewrite_host is not None: - # The port goes with the scheme/host rewrite: one that went with the - # link's original scheme (``http://...:8080``) would otherwise survive - # into an https request and be dialed under TLS. ``userinfo`` goes for - # the reason below -- ``copy_with`` is doing both jobs at once here. + # The port is replaced along with the scheme and host: a port belonging + # to the link's original scheme (``http://...:8080``) would otherwise be kept + # in an https request and be connected to under TLS. ``userinfo`` is dropped for + # the reason below -- ``copy_with`` makes both changes at once here. return str( target.copy_with(scheme="https", host=rewrite_host, port=None, userinfo=b"") ) - # A same-host link may still carry ``user:pass@``, which httpx turns into an + # A same-host link may still contain ``user:pass@``, which httpx turns into an # ``Authorization: Basic`` header on the follow-up request. The host check - # passes in exactly that case, so strip it rather than trust the link. + # passes in that case, so strip it rather than use the link as given. return str(without_embedded_credentials(target)) diff --git a/dataretrieval/transport/liveness.py b/dataretrieval/transport/liveness.py index d72ba3d81..8f2b5faca 100644 --- a/dataretrieval/transport/liveness.py +++ b/dataretrieval/transport/liveness.py @@ -1,15 +1,15 @@ """When data last arrived, shared by the loops that produce and consume it. -A retrieval can be slow for two very different reasons: it is downloading a lot -(progressing, however long it takes) or it is receiving nothing at all (worth giving up -on). Telling those apart needs one fact -- when data last arrived -- that the -page-walking loop knows and the retry loop acts on. Keeping it in this leaf lets -both point *down* at it rather than at each other, and leaves any future producer -of liveness (a streaming body reader, a chunk-level fetch) somewhere to report. - -The stamp lives in a :class:`~contextvars.ContextVar` so concurrent retrievals -- +A retrieval can be slow for two reasons: it is downloading a lot (progressing, +however long it takes) or it is receiving nothing at all (and should be +abandoned). Distinguishing them needs one fact -- when data last arrived -- that the +page-walking loop records and the retry loop reads. Keeping it in this leaf lets +both depend on it rather than on each other, and gives any future producer +of liveness (a streaming body reader, a chunk-level fetch) a place to record it. + +The timestamp is held in a :class:`~contextvars.ContextVar` so concurrent retrievals -- each chunk of a chunked call, each location of a Water Use fan-out -- -measure their own silence instead of sharing one clock. +measure their own time without data instead of sharing one timestamp. """ from __future__ import annotations @@ -23,7 +23,7 @@ def note_progress() -> None: - """Restart the no-progress budget: data just arrived.""" + """Restart the no-progress budget: data has arrived.""" _last_progress.set(time.monotonic()) @@ -34,24 +34,24 @@ def elapsed_since_progress() -> float | None: def credit_wait(seconds: float) -> None: - """Excuse ``seconds`` of sanctioned waiting from the no-progress budget. + """Subtract ``seconds`` of permitted waiting from the no-progress budget. - Two kinds of waiting are not silence: queueing behind a concurrency cap, and - sleeping off a delay the server itself named (see + Two kinds of waiting do not count as time without data: queueing behind a + concurrency cap, and waiting out a delay the server specified (see :meth:`~dataretrieval.transport.retry.RetryPolicy.allows_wait` for why a - sanctioned delay costs the budget nothing). The deep tail of a wide fan-out - can wait past the whole budget and would otherwise start its first attempt - with nothing left to retry with. But neither is progress, and the difference - matters: crediting only the measured wait keeps the budget cumulative across - attempts, where restamping to "now" would also discard silence accumulated - by earlier attempts and quietly turn a bound on total silence into a - per-attempt latency bound. - - The credit never reaches past the present. A wait longer than the whole - budget would otherwise stamp the stamp into the *future*, making - :func:`elapsed_since_progress` negative -- and since nothing ever pulls it - back, that one long queue wait would disable the bound for the rest of the - call, which is precisely the silent-minutes case the budget exists to catch. + server-specified delay is not charged against the budget). The last chunks of a wide + fan-out can wait past the whole budget and would otherwise start their + first attempt with no budget left. Neither is progress, and the + distinction matters: crediting only the measured wait keeps the budget + cumulative across attempts, where resetting the timestamp to now would also discard + time without data accumulated by earlier attempts and turn a bound on total + time without data into a per-attempt latency bound. + + The credit never moves the timestamp past the present. A wait longer than the whole + budget would otherwise set the timestamp in the *future*, making + :func:`elapsed_since_progress` negative -- and since nothing ever reduces + it, that one long queue wait would disable the bound for the rest of the + call, which is the case the budget exists to bound. """ last = _last_progress.get() if last is not None: diff --git a/dataretrieval/transport/pagination.py b/dataretrieval/transport/pagination.py index 3312cb595..dbf97504b 100644 --- a/dataretrieval/transport/pagination.py +++ b/dataretrieval/transport/pagination.py @@ -43,9 +43,9 @@ async def _client_for( client: httpx.AsyncClient | None, ) -> AsyncIterator[httpx.AsyncClient]: - """Borrow a client: the caller's, else the running drive's, else a new one. + """Choose a client: the caller's, else the running fan-out's, else a new one. - Preferring the executor's published client over a fresh one keeps every + Preferring the executor's shared client over a new one keeps every page of every request on one connection pool. """ borrowed = client if client is not None else active_client() @@ -61,7 +61,7 @@ def paginated_failure_message( cause: BaseException, url: str | httpx.URL | None = None, ) -> str: - """Build a recovery-oriented message for an interrupted page walk.""" + """Build a message that names the recovery action for an interrupted page walk.""" cause_str = str(cause).removesuffix(".") if not cause_str.strip(): cause_str = type(cause).__name__ @@ -69,7 +69,7 @@ def paginated_failure_message( action = "wait for the rate-limit window to reset and retry" else: action = "retry the request (possibly after a short backoff)" - # "get a token" is only actionable against the host that honours one. + # The token advice applies only to the host that accepts one. token_advice = ", or obtain an API token" if accepts_api_key(url) else "" return ( f"Paginated request failed after collecting {pages_collected} " @@ -99,7 +99,7 @@ async def paginate( reporter = _progress.current() def report_page(page: httpx.Response, frame: pd.DataFrame) -> None: - note_progress() # a walk still delivering pages is not stalled + note_progress() # a walk still receiving pages is not stalled if reporter is not None: reporter.set_rate_remaining( page.headers.get(_QUOTA_HEADER), @@ -173,14 +173,13 @@ def run_paginated( canonical_url: str | None = None, adapter: str | None = None, ) -> tuple[pd.DataFrame, Any]: - """Drive one full page walk per request through the shared executor. + """Run one full page walk per request through the shared executor. The adapter supplies its strategies (``parse_response``, ``follow_up``, - ``raise_for_status``, and optionally ``finalize``); this driver owns the - composition -- each request paginated on the client the executor publishes - unless ``client`` is injected, the retry - policy, bounded concurrency, and the canonical URL the aggregate reports - (the first request's, unless overridden). + ``raise_for_status``, and optionally ``finalize``); this driver owns the composition + -- each request paginated on the client the executor sets on ``_active_client`` + unless ``client`` is injected, the retry policy, bounded concurrency, and the + canonical URL the aggregate reports (the first request's, unless overridden). Raw transport errors need no mapping in the strategies: the executor retries them and normalizes a deterministic one into the typed diff --git a/dataretrieval/transport/retry.py b/dataretrieval/transport/retry.py index 91af0c956..11f0285ac 100644 --- a/dataretrieval/transport/retry.py +++ b/dataretrieval/transport/retry.py @@ -27,8 +27,8 @@ # Which error statuses a request may be re-sent for. Both are narrower than # :attr:`~dataretrieval.exceptions.DataRetrievalError.retryable`. The default -# keeps every 5xx, for chunked calls riding out a transient upstream failure; -# the gateway-only set is for the single-shot adapters whose service answers a +# keeps every 5xx, for chunked calls that retry through a transient upstream failure; +# the gateway-only set is for the single-shot adapters whose service responds to a # *rejected query* with a 500 -- WQP for an over-large request, StreamStats for # out-of-network coordinates. Which failures may be re-sent is ADR 0006. _RETRYABLE_STATUSES = frozenset({429, *range(500, 600)}) @@ -36,11 +36,11 @@ _RETRY_BASE_BACKOFF = 0.5 _RETRY_MAX_BACKOFF = 30.0 _RETRY_AFTER_CAP = 60.0 -# Most a server-named delay is nudged by, to keep chunks handed the same -# hint from waking together. Small on purpose: the server named the wait, so -# jitter here decorrelates rather than extends it. +# Most a server-named delay is extended by, to keep chunks given the same +# value from retrying together. Small deliberately: the server specified the wait, so +# jitter here only decorrelates the retries. _RETRY_AFTER_JITTER = 1.0 -# Attempts the no-progress budget never withholds; see RetryPolicy.allows_wait. +# Attempts the no-progress budget never blocks; see RetryPolicy.allows_wait. _STALL_EXEMPT_ATTEMPTS = 1 _T = TypeVar("_T") @@ -50,21 +50,21 @@ class RetryPolicy: """Immutable bounded exponential-backoff-with-full-jitter policy. - Two independent bounds decide when to stop: :attr:`max_retries` caps *how - many* attempts a failure gets, and :attr:`stall_timeout` caps *how long* a + Two independent bounds determine when retrying stops: :attr:`max_retries` caps how + many attempts are made after a failure, and :attr:`stall_timeout` caps *how long* a call may go on receiving nothing. """ #: Attempts after the first. ``0`` disables retry entirely. The default is - #: ``config``'s, not a second copy of it: a directly-constructed policy and - #: one built by :meth:`from_configuration` must agree on the retry budget. + #: read from ``configuration``, not copied: a directly-constructed policy and + #: one built by :meth:`from_configuration` must use the same retry budget. max_retries: int = _configuration.DEFAULT_RETRIES #: First backoff ceiling; doubles per attempt up to :attr:`max_backoff`. base_backoff: float = _RETRY_BASE_BACKOFF - #: Ceiling for our own exponential backoff. + #: Ceiling for the client's exponential backoff. max_backoff: float = _RETRY_MAX_BACKOFF - #: Longest server-named ``Retry-After`` we are willing to wait out inline. - #: A longer one stops the retry and surfaces a resumable transient, so the + #: Longest server-named ``Retry-After`` that is waited out inline. + #: A longer one stops the retry and raises a resumable transient, so the #: caller decides whether to wait rather than blocking inside the request. retry_after_cap: float = _RETRY_AFTER_CAP #: Error statuses this policy will re-send for. Defaults to 429 and every @@ -72,16 +72,16 @@ class RetryPolicy: #: pass :data:`_GATEWAY_STATUSES` instead. retryable_statuses: frozenset[int] = _RETRYABLE_STATUSES #: Longest a call may go *without receiving any data* before retrying stops - #: and the failure surfaces -- the total of every silent attempt and every - #: unsanctioned wait since the last page arrived (a server-named - #: ``Retry-After`` and time queued behind the concurrency gate are excused; - #: see :meth:`allows_wait`). Bounds the wall-clock cost of a dead - #: connection or a service that keeps refusing, which :attr:`max_retries` + #: and the failure is raised -- the total of every attempt that received + #: nothing and every uncredited wait since the last page arrived (a + #: server-named ``Retry-After`` and time queued behind the concurrency gate + #: are credited back; see :meth:`allows_wait`). Bounds the wall-clock cost of a dead + #: connection or a service that keeps returning errors, which :attr:`max_retries` #: alone does not: it counts attempts, not seconds, so four retries of a - #: request that times out after a minute is four silent minutes. Progress - #: resets the clock (see + #: request that times out after a minute is four minutes without data. + #: Progress restarts the budget (see #: :func:`~dataretrieval.transport.liveness.note_progress`), so a slow but - #: productive download is never cut short, and an attempt already in flight + #: productive download is never stopped early, and an attempt already in flight #: is never interrupted. ``0`` disables the bound. See :meth:`allows_wait` #: for how it is applied. stall_timeout: float = _configuration.DEFAULT_STALL_TIMEOUT @@ -112,8 +112,8 @@ def from_configuration( :mod:`dataretrieval.configuration` -- a ``configure()`` block, then the environment variable, then the config file. ``adapter`` names the adapter this policy is for, so a ``[wqp] retries = 2`` table applies to - WQP calls and nothing else; ``None`` resolves package-wide. The pure - timing knobs stay module constants read at call time so a test's + WQP calls and nothing else; ``None`` resolves package-wide. The + timing values stay module constants read at call time so a test's ``monkeypatch.setattr`` still applies. """ statuses = ( @@ -129,7 +129,7 @@ def from_configuration( ) def should_retry(self, attempt: int, retry_after: float | None) -> bool: - """Whether a just-failed 1-based attempt warrants another try.""" + """Whether another attempt is allowed after a failed 1-based attempt.""" if attempt > self.max_retries: return False return retry_after is None or retry_after <= self.retry_after_cap @@ -143,30 +143,30 @@ def allows_wait( ) -> bool: """Whether waiting ``delay`` more fits the no-progress budget. - ``elapsed`` is the silence so far (see + ``elapsed`` is the time without data so far (see :func:`~dataretrieval.transport.liveness.elapsed_since_progress`), passed in rather than read here so the policy stays a pure value object. The first retry is always allowed. One slow attempt can spend the whole - budget on its own -- a heavy page against a loaded service, or any + budget on its own -- a large page from a busy service, or any attempt that runs to the read timeout -- and letting that suppress retry entirely would turn a recoverable transient into an immediate failure - for exactly the large queries that most need retrying. So the budget - bounds *repeated* silence: with the defaults a dead connection costs - about two read timeouts rather than five attempts' worth. - - A delay the *server* named -- ``retry_after`` is not ``None``, the same - hint :meth:`should_retry` and :meth:`backoff` take -- costs the budget - nothing. Charging for it would mean a service that answers 429 with - ``Retry-After: 30`` gets fewer retries than one that says nothing at all + for the large queries where retrying matters most. So the budget + bounds *repeated* time without data: with the defaults a dead connection costs + about two read timeouts rather than five attempts. + + A delay the *server* specified -- ``retry_after`` is not ``None``, the same + value :meth:`should_retry` and :meth:`backoff` take -- is not charged + against the budget. Charging for it would mean a service that responds 429 with + ``Retry-After: 30`` gets fewer retries than one that sends no ``Retry-After`` -- with the shipped defaults (a 60 s budget, a 60 s - :attr:`retry_after_cap`) any honored hint of half the budget or more - would allow exactly one retry no matter what - :attr:`max_retries` says. Waiting because we were told to is not the - service going quiet on us; it is the service telling us when to come - back. The driver credits the same wait back afterwards (see + :attr:`retry_after_cap`) any accepted value of half the budget or more + would allow one retry regardless of + :attr:`max_retries`. A wait the server specified is not time without a + response; the service has specified when to retry. The driver credits the + same wait back afterwards (see :func:`~dataretrieval.transport.liveness.credit_wait`) so it doesn't - accumulate into the *next* attempt's silence either. + accumulate into the *next* attempt's time without data either. """ if attempt <= _STALL_EXEMPT_ATTEMPTS: return True @@ -179,21 +179,21 @@ def allows_wait( def backoff(self, attempt: int, retry_after: float | None) -> float: """Seconds to wait before a 1-based retry attempt. - A jittered component is always included, even when the server named a - delay: a hint of ``0`` -- or a ``Retry-After`` date that has already + A jittered component is always included, even when the server specified a + delay: a value of ``0`` -- or a ``Retry-After`` date that has already passed -- would otherwise become a zero-delay re-send against a service - that just asked us to slow down, and chunks handed the same hint - would all wake at the same instant and burst together. + that just sent a ``Retry-After``, and chunks given the same value + would all retry at the same instant. - On a server hint that jitter is a small decorrelating nudge rather than + On a server value that jitter is a small decorrelating offset rather than a second backoff, and the total is held to :attr:`retry_after_cap`: - full jitter on top of a hint already at the cap would sleep half again - as long as any bound this policy declares. It is bounded by + full jitter on top of a value already at the cap would wait up to 1.5 + times the longest bound this policy declares. It is bounded by :attr:`max_backoff` rather than by this attempt's exponential ceiling, - so it survives a :attr:`base_backoff` of zero -- the case where the - ceiling collapses and a hint of ``0`` would otherwise become exactly the - zero-delay re-send this prevents. A policy that declares no backoff at - all still gets none. + so it still applies when :attr:`base_backoff` is zero -- the case where the + ceiling is zero and a value of ``0`` would otherwise become the + zero-delay re-send this prevents. A policy with no backoff configured + still applies none. """ ceiling = min(self.max_backoff, self.base_backoff * 2 ** (attempt - 1)) if retry_after is None: @@ -208,7 +208,7 @@ def backoff(self, attempt: int, retry_after: float | None) -> float: def _retryable( exc: BaseException, statuses: frozenset[int] = _RETRYABLE_STATUSES ) -> tuple[bool, float | None]: - """Return whether ``exc`` is safe to retry and any server delay hint.""" + """Return whether ``exc`` is safe to retry and any server-specified delay.""" if isinstance(exc, TransientError): if exc.status_code is not None and exc.status_code not in statuses: return False, None @@ -219,11 +219,11 @@ def _retryable( class _Wait(NamedTuple): - """How long to hold off before a retry, and whether the server asked for it. + """How long to wait before a retry, and whether the server specified the delay. - ``sanctioned`` travels with the delay because only the driver knows when the - sleep finished, and a server-named wait has to be credited back to the - no-progress budget once it has been served (see + ``sanctioned`` is stored with the delay because only the driver observes when + the sleep finished, and a server-named wait has to be credited back to the + no-progress budget once it has elapsed (see :meth:`RetryPolicy.allows_wait`). """ @@ -231,12 +231,13 @@ class _Wait(NamedTuple): sanctioned: bool def settle(self) -> None: - """Credit a served server-named wait back to the no-progress budget. + """Credit an elapsed server-specified wait back to the no-progress budget. Paired with the sleep rather than left to each driver: a wait that - :meth:`RetryPolicy.allows_wait` excused going in has to be excused coming - out too, or it accumulates into the *next* attempt's silence and caps the - retries anyway. Both drivers sleep differently but settle identically. + :meth:`RetryPolicy.allows_wait` did not charge must also be credited + back afterwards, or it accumulates into the *next* attempt's time + without data and caps the retries anyway. Both drivers call this after + their own sleep. """ if self.sanctioned: credit_wait(self.delay) @@ -266,11 +267,12 @@ async def retry_async( ``gate`` bounds how many attempts run concurrently. Owning it here rather than letting each caller wrap its own body keeps two rules in one place: the - slot is acquired per *attempt*, so a call sleeping off a backoff isn't - holding one while it isn't touching the server, and the time spent waiting + slot is acquired per *attempt*, so a call waiting out a backoff does not + hold one while it is not making a request, and the time spent waiting for it is credited back to the no-progress budget rather than counted as - silence. A caller that gated its own body would have to rediscover both, and - nothing would flag a caller that broke either. + time without data. A caller that gated its own body would have to + re-implement both, and + nothing would detect a caller that broke either. """ policy = RetryPolicy.from_configuration() if policy is None else policy attempt = 0 diff --git a/dataretrieval/utils.py b/dataretrieval/utils.py index 0eda32048..2ae465b5b 100644 --- a/dataretrieval/utils.py +++ b/dataretrieval/utils.py @@ -1,8 +1,8 @@ """Data-shaping helpers, plus re-exports for names documented at this path. -What is *defined* here is frame munging that names no service: building a UTC -datetime column out of the separate date/time/zone columns a caller points at. -The one-shot HTTP query path lives in :mod:`dataretrieval._querying` and the +What is *defined* here is frame reshaping that names no service: building a UTC +datetime column out of the separate date/time/zone columns a caller names. +The one-shot HTTP query path is in :mod:`dataretrieval._querying` and the WQX3 / legacy-WQP column conventions in :mod:`dataretrieval._wqx`; nothing here depends on either -- the names below are re-exported so their documented ``dataretrieval.utils`` paths keep resolving. diff --git a/dataretrieval/waterdata/configuration.py b/dataretrieval/waterdata/configuration.py index 356d10323..eb3535705 100644 --- a/dataretrieval/waterdata/configuration.py +++ b/dataretrieval/waterdata/configuration.py @@ -1,10 +1,9 @@ """The settings the Water Data adapter reads -- its configuration profile. -A file of its own because :mod:`dataretrieval.waterdata` is a package rather -than a single module; every other adapter declares its class in the module a -caller imports. Either way the point is the same: a setting's definition sits -with the code that reads it, so adding one no longer edits a service-neutral -file (ADR 0011). +A file of its own because :mod:`dataretrieval.waterdata` is a package rather than a +single module; every other adapter declares its class in the module a caller imports. +Either way the point is the same: a setting's definition is in the module that reads it, +so adding one no longer edits a service-neutral file (ADR 0011). """ from __future__ import annotations @@ -44,12 +43,11 @@ class WaterdataConfiguration( stall_timeout : float, optional Seconds a call may go without receiving any data before retrying stops. base_url : str, optional - Root to send Water Data requests to, instead of the service's own. The - package appends its own paths, so one value moves all four families - together -- ``/ogcapi/v0``, ``/samples-data``, ``/statistics/v0`` and - ``/stac/v0``. Code only: the file and the environment refuse it. The - API key is scoped to the host that honors it, so a redirected call - carries no key. + Root to send Water Data requests to, instead of the service's own. The package + appends its own paths, so one value redirects all four families together -- + ``/ogcapi/v0``, ``/samples-data``, ``/statistics/v0`` and ``/stac/v0``. Code + only: the file and the environment refuse it. The API key is scoped to the host + that accepts it, so a redirected call sends no key. concurrency : int or str, optional Cap on simultaneous sub-requests, or ``"unbounded"``. parallel_chunks : int, optional @@ -58,11 +56,11 @@ class WaterdataConfiguration( """ # The settings this service reads, named by the groups they come from: - # every adapter's retry dials, a redirectable base, and -- because Water + # every adapter's retry settings, a redirectable base, and -- because Water # Data queries divide along a URL byte budget and are executed concurrently - # -- both fan-out dials. Each group declares the setting itself once, in - # :mod:`dataretrieval.configuration`, which is also where its grammar and - # its coercion live. + # -- both fan-out settings. Each group declares the setting itself once, in + # :mod:`dataretrieval.configuration`, which also defines its grammar and + # its coercion. adapter: ClassVar[str] = "waterdata" diff --git a/dataretrieval/waterdata/cql.py b/dataretrieval/waterdata/cql.py index 5cc5081c5..0fdef64f2 100644 --- a/dataretrieval/waterdata/cql.py +++ b/dataretrieval/waterdata/cql.py @@ -1,9 +1,9 @@ """One getter for queries the typed getters cannot express. The other families expose a fixed argument per filter, which covers the common -cases and keeps them discoverable. This is the escape hatch: an arbitrary CQL2 -filter against any collection, for the query nobody anticipated. Prefer a typed -getter when one fits -- it validates more and names its filters. +cases and keeps them discoverable. This is the generic path: an arbitrary CQL2 +filter against any collection, for a query the typed getters do not cover. +Prefer a typed getter when one fits -- it validates more and names its filters. """ from __future__ import annotations @@ -76,7 +76,7 @@ def get_cql( (e.g. ``"daily"``, ``"monitoring-locations"``). cql : str or dict CQL2 query. A ``dict`` is JSON-serialized for transport; a ``str`` is - sent through unchanged. The query goes into the HTTP POST body with + sent through unchanged. The query is sent in the HTTP POST body with ``Content-Type: application/query-cql-json``. properties : str or iterable of str, optional Server-side property whitelist (passed as ``properties=`` on the URL). @@ -159,16 +159,15 @@ def get_cql( ), ) - # ``dict`` is the pythonic input — serialize on the way out. ``str`` is sent + # A ``dict`` is serialized before sending. ``str`` is sent # verbatim so callers who already have a CQL2 doc (e.g. imported from a # config file) don't need to re-parse it. body = json.dumps(cql, separators=(",", ":")) if isinstance(cql, dict) else cql - # The engine owns the rest — the wire-properties id-switch, request - # construction, pagination, and finalization — behind the same Water Data - # entry the typed getters use; ``cql_body`` selects the verbatim-CQL2 - # shape. ``output_id`` defaults from the collection map, which the guard - # above has already confirmed covers ``collection``. + # The engine handles the wire-properties id-switch, request construction, + # pagination, and finalization, through the same Water Data entry the typed getters + # use; ``cql_body`` selects the verbatim-CQL2 shape. ``output_id`` defaults from the + # collection map, which the check above has already confirmed covers ``collection``. args = _get_args( { "properties": properties, diff --git a/dataretrieval/waterdata/endpoints.py b/dataretrieval/waterdata/endpoints.py index c9efea66d..8c153b872 100644 --- a/dataretrieval/waterdata/endpoints.py +++ b/dataretrieval/waterdata/endpoints.py @@ -1,8 +1,8 @@ -"""Every Water Data endpoint this package talks to, in one place. +"""Every Water Data endpoint this package requests, in one place. -The host is the authority of the credentials leaf -- the host that serves -these endpoints is the host that honors the API key -- while the paths below -stay here rather than importing OGC policy internals. This module imports only +The credentials leaf defines the host -- the host that serves these +endpoints is the host that accepts the API key -- and the paths below are +defined here rather than importing OGC policy internals. This module imports only leaves: the credentials host and the configuration chain (ADR 0003). """ diff --git a/dataretrieval/waterdata/measurements.py b/dataretrieval/waterdata/measurements.py index fe5d5acc8..5060a2677 100644 --- a/dataretrieval/waterdata/measurements.py +++ b/dataretrieval/waterdata/measurements.py @@ -154,7 +154,7 @@ def get_field_measurements( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -286,9 +286,8 @@ def get_peaks( Calendar / water-year filters on the peak event. The water year ends September 30 (e.g. WY2024 = Oct 1, 2023 – Sep 30, 2024). peak_since : int or list of ints, optional - Filter on the year since which the peak value has stood as the - record (the API serves this field as an integer; many rows are - ``null``). + Filter on the year since which the peak value has been the record (the API + serves this field as an integer; many rows are ``null``). properties : string or iterable of strings, optional Subset of columns to return. Defaults to every available property. skip_geometry : boolean, optional @@ -517,7 +516,7 @@ def get_channel( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. diff --git a/dataretrieval/waterdata/metadata.py b/dataretrieval/waterdata/metadata.py index 625cefdfe..eb26d98e5 100644 --- a/dataretrieval/waterdata/metadata.py +++ b/dataretrieval/waterdata/metadata.py @@ -1,8 +1,8 @@ -"""Getters that answer "what data exists?" rather than returning it. +"""Getters that describe what data exists rather than returning it. The monitoring-location catalog, the time-series inventory, and the joins over them. These are the discovery step: narrow down which locations and parameters -are worth requesting before pulling observations from +to request before pulling observations from :mod:`~dataretrieval.waterdata.time_series`. """ @@ -276,7 +276,7 @@ def get_monitoring_locations( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -531,7 +531,7 @@ def get_time_series_metadata( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -666,8 +666,8 @@ def get_combined_metadata( The ``combined-metadata`` collection joins the monitoring-locations catalog with the time-series-metadata catalog so that one row is returned per (location, parameter, statistic) inventory entry, - carrying every column from both source endpoints. This makes it the - most flexible "what data is available" endpoint in the Water Data + with every column from both source endpoints. This makes it the + most flexible inventory endpoint in the Water Data API: any monitoring-location attribute (state, HUC, site type, drainage area, well-construction depth, …) can be combined with any time-series attribute (parameter code, statistic, data type, period @@ -711,8 +711,8 @@ def get_combined_metadata( Indicates whether the data from this time series represent a specific statistical computation. thresholds : number or list of numbers, optional - Numeric limits known for a time series (e.g. historic maximum, - below-which-the-sensor-is-non-operative). + Numeric limits known for a time series (e.g. the historic maximum, or the level + below which the sensor is non-operative). sublocation_identifier : string or iterable of strings, optional primary : string or iterable of strings, optional A flag identifying whether the time series is "primary". Primary @@ -735,7 +735,7 @@ def get_combined_metadata( two-digit ANSI/FIPS code (``"55"``). state_name, county_name, hydrologic_unit_code, site_type, \ site_type_code : string or iterable of strings, optional - Common location-catalog filters carried over from the + Common location-catalog filters shared with the ``monitoring-locations`` collection. The function also accepts the full list of location-catalog kwargs (agency, district, altitude, vertical/horizontal datum, drainage area, aquifer, @@ -827,7 +827,7 @@ def get_combined_metadata( ... parameter_code="00060", ... ) - >>> # Two-step "what's available?" → "fetch it" workflow: + >>> # Two-step workflow: inventory, then fetch: >>> # 1. inventory the monitoring locations in two HUCs >>> hucs, _ = dataretrieval.waterdata.get_combined_metadata( ... hydrologic_unit_code=["11010008", "11010009"], @@ -879,9 +879,8 @@ def get_field_measurements_metadata( This is the discrete-measurement analogue to :func:`get_time_series_metadata` (which describes daily and continuous - series). It's primarily useful for inventory queries: "what - field-measurement parameters does this site have, and over what date - range?" + series). It is primarily useful for inventory queries: which + field-measurement parameters a site has, and over what date range. See the OpenAPI reference for the full list of supported fields: https://api.waterdata.usgs.gov/ogcapi/v0/openapi?f=html#/field-measurements-metadata diff --git a/dataretrieval/waterdata/nearest.py b/dataretrieval/waterdata/nearest.py index ea91fab1b..666d57c36 100644 --- a/dataretrieval/waterdata/nearest.py +++ b/dataretrieval/waterdata/nearest.py @@ -1,4 +1,4 @@ -"""``get_nearest_continuous``: nearest-timestamp convenience on top of +"""``get_nearest_continuous``: nearest-timestamp selection over ``get_continuous``. Built on the CQL ``filter`` passthrough; only ``get_nearest_continuous`` is public — everything else is package-private. """ @@ -118,17 +118,17 @@ def get_nearest_continuous( that falls in any window. Then, per ``(monitoring_location_id, target)`` pair, picks the single observation with the smallest ``|time - target|``. - The USGS continuous endpoint matches ``time`` parameters exactly rather - than fuzzily, and it does not implement ``sortby`` for arbitrary fields; - this function is the single-round-trip way to ask "what reading is - nearest this timestamp?" for many timestamps at once. + The USGS continuous endpoint matches ``time`` parameters exactly, with + no tolerance, and it does not implement ``sortby`` for arbitrary fields; + this function finds the reading nearest each of many timestamps in one + round trip. Parameters ---------- targets : list-like of datetime-convertible Target timestamps. Naive datetimes are treated as UTC. Accepts a list, ``pandas.Series``, ``pandas.DatetimeIndex``, ``numpy.ndarray``, - or anything ``pandas.to_datetime`` consumes. + or anything ``pandas.to_datetime`` accepts. monitoring_location_id : string or iterable of strings, optional Forwarded to ``get_continuous``. parameter_code : string or iterable of strings, optional @@ -143,11 +143,10 @@ def get_nearest_continuous( `_ for the full grammar. - Must be small enough that every target's window captures + Must be small enough that every target's window contains roughly one observation at the service cadence. The default matches a 15-minute continuous gage; widen (e.g. - ``"PT15M"``) for irregular cadences or resilience to data - gaps. + ``"PT15M"``) for irregular cadences or tolerance of data gaps. on_tie : {"first", "last", "mean"}, default ``"first"`` How to resolve ties when two observations are exactly equidistant from a target (which happens when the target falls at the midpoint @@ -160,14 +159,13 @@ def get_nearest_continuous( the target, since no real observation exists at the midpoint. **kwargs - Additional keyword arguments forwarded to ``get_continuous`` - (e.g. ``statistic_id``, ``approval_status``, ``properties``). - Passing ``time``, ``filter``, or ``filter_lang`` raises - ``TypeError`` — this function builds those itself. A caller-provided - ``properties`` list gains ``time`` and ``monitoring_location_id`` when - either is omitted: the match is computed against the first and grouped - by the second, so the returned frame carries both columns even when they - were not requested. + Additional keyword arguments forwarded to ``get_continuous`` (e.g. + ``statistic_id``, ``approval_status``, ``properties``). Passing ``time``, + ``filter``, or ``filter_lang`` raises ``TypeError`` — this function builds those + itself. A caller-provided ``properties`` list has ``time`` and + ``monitoring_location_id`` when either is omitted: the match is computed against + the first and grouped by the second, so the returned frame includes both columns + even when they were not requested. Returns ------- @@ -176,7 +174,7 @@ def get_nearest_continuous( had at least one observation in its window. Rows are augmented with a ``target_time`` column indicating which target they correspond to. Targets with no observations in their window are - silently dropped. + omitted; no warning is raised. md : :class:`~dataretrieval.utils.BaseMetadata` Metadata from the underlying ``get_continuous`` call. @@ -191,17 +189,17 @@ def get_nearest_continuous( ----- *Window sizing and ties.* When ``window`` is exactly half the service cadence, most targets' windows contain a single observation and - ``on_tie`` is moot. Ties arise only when a target sits exactly at the + ``on_tie`` has no effect. Ties arise only when a target falls exactly at the midpoint between two grid observations — rare in practice but possible. Setting ``window`` to a full cadence (or larger) guarantees at least one observation per target in steady state at the cost of more bytes per response. - *Why windowed CQL rather than sort+limit.* The API's advertised - ``sortby`` parameter would make this a one-liner per target (``filter`` - by ``time <= t`` and ``limit 1``), but it is per-query — you would need - one HTTP round-trip per target. The CQL ``OR``-chain approach folds - all N targets into one request (auto-chunked when the URL is long). + *Why windowed CQL rather than sort+limit.* The API's documented ``sortby`` parameter + would make this a trivial query per target (``filter`` by ``time <= t`` and ``limit + 1``), but it is per-query — you would need one HTTP round-trip per target. The CQL + ``OR``-chain approach combines all N targets into one request (auto-chunked when the + URL is long). Examples -------- @@ -313,7 +311,7 @@ def _select_nearest_rows( def _coerce_targets(targets: Any) -> pd.DatetimeIndex: - """Accept anything ``pandas.to_datetime`` consumes, including a single value. + """Accept anything ``pandas.to_datetime`` accepts, including a single value. A bare scalar (string, ``Timestamp``, ``datetime``, …) becomes a one-element ``DatetimeIndex``; an iterable (list, ``Series``, ``ndarray``) @@ -326,7 +324,7 @@ def _coerce_targets(targets: Any) -> pd.DatetimeIndex: def _check_nearest_kwargs(kwargs: dict[str, Any], on_tie: OnTie) -> None: - """Reject kwargs the helper owns; validate ``on_tie``.""" + """Reject kwargs the helper sets itself; validate ``on_tie``.""" for forbidden in ("time", "filter", "filter_lang"): if forbidden in kwargs: raise TypeError( @@ -361,7 +359,7 @@ def _pick_nearest_row( """Return the single row within ``window_td`` of ``target``, or ``None``. Resolves ties (two rows equidistant from ``target``) per ``on_tie``. - The returned row carries a ``target_time`` column identifying which + The returned row includes a ``target_time`` column identifying which target it was selected for. """ in_window = site_df[ diff --git a/dataretrieval/waterdata/ratings.py b/dataretrieval/waterdata/ratings.py index 655dd16fa..7a3f2bce0 100644 --- a/dataretrieval/waterdata/ratings.py +++ b/dataretrieval/waterdata/ratings.py @@ -95,8 +95,7 @@ def get_ratings( download_and_parse : bool, default ``True`` If ``True``, download every matching RDB file and parse it into a ``DataFrame``. If ``False``, return the raw list of STAC feature - dicts so the caller can inspect what's available before pulling - bytes. + dicts so the caller can inspect what's available before downloading. ssl_check : bool, default ``True`` Verify the server's SSL certificate. @@ -105,7 +104,7 @@ def get_ratings( dict[str, pandas.DataFrame] or list[dict] When ``download_and_parse=True`` (the default), a dict keyed by feature ID (e.g. ``"USGS-01104475.exsa.rdb"``) mapping to a parsed - ``DataFrame``. Each frame carries provenance in + ``DataFrame``. Each frame records provenance in ``df.attrs["comment"]`` (the RDB ``#``-prefixed header lines, like rating id, parameter, last-shifted timestamp) and ``df.attrs["url"]`` (the asset URL it was fetched from). When @@ -123,24 +122,22 @@ def get_ratings( or :class:`~dataretrieval.exceptions.NetworkError` if a request can't reach the service in a way retrying cannot fix. FanOutInterrupted - A transient failure (429 / 5xx / timeout) survived the built-in - retries during the search or a download. ``exc.call.resume()`` - finishes the interrupted stage (see :doc:`/userguide/errors`); the - assembled per-feature dict is returned by a fresh ``get_ratings`` - call. + A transient failure (429 / 5xx / timeout) was not resolved by the built-in + retries during the search or a download. ``exc.call.resume()`` finishes the + interrupted stage (see :doc:`/userguide/errors`); the assembled per-feature dict + is returned by a new ``get_ratings`` call. Warns ----- SkippedRatingWarning - One feature of the batch failed *deterministically* -- a stale - catalog entry (404 on its data asset), a feature with no data asset, - a malformed RDB file. That feature is skipped and its id is absent - from the returned dict; the rest of the batch is unaffected. A - monitoring location with no published rating never warns -- it - matches no feature in the search, so there is nothing to skip. See + One feature of the batch failed *deterministically* -- a stale catalog entry + (404 on its data asset), a feature with no data asset, a malformed RDB file. + That feature is skipped and its id is absent from the returned dict; the rest of + the batch is unaffected. A monitoring location with no published rating never + produces a warning -- it matches no feature in the search, so there is nothing + to skip. See :class:`~dataretrieval.exceptions.SkippedItemWarning` for the policy - (transients never skip) and the ``filterwarnings`` recipe that makes - a skip fatal. + (transients never skip) and the ``filterwarnings`` call that makes a skip fatal. Examples -------- @@ -185,7 +182,7 @@ def get_ratings( else None ) - # Mirror R: pin file_type server-side only when one type is requested. + # Match R: set file_type server-side only when one type is requested. server_file_type = file_types[0] if len(file_types) == 1 else None filter_str = _build_filter(monitoring_location_id, server_file_type) @@ -263,14 +260,14 @@ def _search( limit: int, ssl_check: bool, ) -> list[dict[str, Any]]: - """Run STAC ``/search`` and return ALL matching features. + """Run STAC ``/search`` and return all matching features. ``limit`` is the page size (clamped to the service maximum of 10,000); the STAC ``next`` link is followed until exhausted so a result set larger than - one page isn't silently truncated. + one page is not truncated. The page walk is :func:`~dataretrieval.transport.pagination.run_paginated` - with STAC strategies. Pages carry features rather than rows, so each page + with STAC strategies. Pages hold features rather than rows, so each page frame wraps the raw feature dicts in a single ``feature`` column. """ query_params: dict[str, Any] = {"limit": min(limit, 10000)} @@ -287,13 +284,13 @@ def _search( def parse_response(resp: httpx.Response) -> tuple[pd.DataFrame, str | None]: body = resp.json() page = pd.DataFrame({"feature": body.get("features", [])}) - # The STAC ``next`` link is a fully-formed GET href carrying the + # The STAC ``next`` link is a complete GET href that includes the # limit/filter/bbox and a continuation token, so it becomes the # cursor verbatim -- except for the shared safety policy: the href is # response data, so it is checked before it becomes a request. A link - # to another host would carry this request's API key off the - # authorized host, and one carrying ``user:pass@`` would mint an - # ``Authorization: Basic`` header the caller never configured. + # to another host would send this request's API key to a host + # other than the authorized one, and one with ``user:pass@`` would + # produce an ``Authorization: Basic`` header the caller never configured. href = next( (lnk["href"] for lnk in body.get("links", []) if lnk.get("rel") == "next"), None, @@ -331,10 +328,10 @@ def _inert_response( ) -> httpx.Response: """A body-less stand-in the executor can aggregate. - The executor keeps every completed item's response until the drive ends, + The executor keeps every completed item's response until the run ends, but its aggregation reads only status, headers, and URL -- never the - body. Handing it a stand-in keeps a large batch from pinning every - downloaded file in memory for the whole drive. ``elapsed`` is left + body. Passing it a stand-in keeps a large batch from holding every + downloaded file in memory for the whole run. ``elapsed`` is left unset; the aggregate's ``_safe_elapsed`` treats that as zero. """ return httpx.Response(status, headers=headers, request=httpx.Request("GET", url)) @@ -345,10 +342,9 @@ async def _fetch_rating( ) -> tuple[pd.DataFrame, httpx.Response]: """Fetch one feature's data asset, parse RDB, optionally persist to disk. - Headers are evaluated against each asset href -- assets can live on a - different host than the catalog, and must not inherit its auth. Runs - inside a drive: the executor publishes the shared client before any - fetch starts. + Headers are evaluated against each asset href -- assets can be on a different host + than the catalog, and must not receive its credentials. Runs inside a fan-out run: + the executor sets the shared client before any fetch starts. """ fid = feature["id"] href = _asset_href(feature) @@ -361,7 +357,7 @@ async def _fetch_rating( headers = _default_headers(href) session = active_client() if session is None: - raise RuntimeError("_fetch_rating must run inside a FanOut drive.") + raise RuntimeError("_fetch_rating must run inside a FanOut run.") response = await session.get(href, headers=headers) _raise_for_non_200(response) @@ -382,7 +378,7 @@ def _download_all( ) -> dict[str, pd.DataFrame]: """Download every feature's rating over the shared fan-out executor. - The plan is the feature list itself -- ``FanOut`` asks a plan only to be + The plan is the feature list itself -- ``FanOut`` requires a plan only to be sized and iterable -- so the downloads get bounded concurrency, per-attempt retry, the progress line, and the resumable interruption taxonomy. @@ -394,15 +390,15 @@ def _download_all( warns with :class:`~dataretrieval.exceptions.SkippedRatingWarning` and skips the feature. ``OSError`` writing ``file_path`` propagates -- a local disk problem is not a per-feature condition. Raw ``httpx`` errors - pass through untouched so the executor can classify and retry them. + pass through unchanged so the executor can classify and retry them. The public result is a dict keyed by feature id, so the fetch closure accumulates it; the executor's combined frame is not the return shape and - is discarded. Both outcomes hand the executor a body-less + is discarded. Both outcomes return to the executor a body-less :func:`_inert_response` -- a skip so the item counts as complete (a later ``resume()`` continues past it rather than re-attempting), a success so - the drive doesn't pin every downloaded file in memory while keeping the - real status and quota headers for aggregation. + the run does not hold every downloaded file in memory while keeping the + response's status and quota headers for aggregation. """ out: dict[str, pd.DataFrame] = {} if not features: diff --git a/dataretrieval/waterdata/reference.py b/dataretrieval/waterdata/reference.py index 8791b5576..fc790052f 100644 --- a/dataretrieval/waterdata/reference.py +++ b/dataretrieval/waterdata/reference.py @@ -2,7 +2,7 @@ Reference tables and per-collection queryables -- the parameter codes, statistic codes, and filterable properties the other getters accept. These describe the -service rather than the water, so they are the one family whose results are +service rather than the measurements, so they are the one family whose results are mostly stable between calls. """ @@ -48,7 +48,7 @@ def get_reference_table( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. + number if your internet connection is unreliable. query: dictionary, optional A dictionary of extra query parameters to pass to the collection API call. @@ -63,7 +63,7 @@ def get_reference_table( ------- df : ``pandas.DataFrame`` or ``geopandas.GeoDataFrame`` Formatted data returned from the API query. The primary metadata - of each reference table will show up in the first column, where + of each reference table appears in the first column, where the name of the column is the singular form of the collection name, separated by underscores (e.g. the "medium-codes" reference table has a column called "medium_code", which contains all possible @@ -120,7 +120,7 @@ def get_queryables(collection: str) -> tuple[pd.DataFrame, BaseMetadata]: """List the queryable properties of a Water Data API collection. Every OGC collection (``daily``, ``continuous``, ``monitoring-locations``, - ...) advertises the set of properties that can be filtered on -- exposed as + ...) lists the set of properties that can be filtered on -- exposed as the typed keyword arguments of the matching ``get_*`` function, and usable directly in a CQL2 ``filter``. This function returns that set, so you can discover the available filters programmatically and monitor them for @@ -160,9 +160,9 @@ def get_queryables(collection: str) -> tuple[pd.DataFrame, BaseMetadata]: 'string' """ # Reading the queryables document is OGC protocol work; this getter only - # names the API to ask -- which is the redirected one when a ``configure`` + # names the API to query -- which is the redirected one when a ``configure`` # block set a base URL, so the queryables describe the API the getters are - # actually querying. + # querying. return queryables_frame(collection, base_url=ogc_api_url()) diff --git a/dataretrieval/waterdata/samples.py b/dataretrieval/waterdata/samples.py index e8e024c88..6b07a5e45 100644 --- a/dataretrieval/waterdata/samples.py +++ b/dataretrieval/waterdata/samples.py @@ -3,7 +3,7 @@ Discrete water-quality results, which come from a different upstream service than the rest of Water Data -- with its own parameter spellings and its own error envelope. The translation between this package's argument names and that -service's wire names lives here, next to the getters that need it. +service's wire names is defined here, next to the getters that need it. """ from __future__ import annotations @@ -85,7 +85,7 @@ def _get_samples_csv( ) -> tuple[pd.DataFrame, httpx.Response]: """Issue a Samples CSV request and parse the body into a DataFrame. - Shared tail for the Samples getters: sends the GET with the standard + Shared final step for the Samples getters: sends the GET with the standard headers (including ``X-Api-Key``), raises a typed error on a non-200 (consistent with the OGC/stats path) instead of a bare ``HTTPStatusError``, and reads the CSV. The caller wraps the response @@ -107,8 +107,8 @@ def _get_samples_csv( # Map the public snake_case ``get_samples`` parameters to the camelCase query # parameter names the Samples API expects on the wire. ``characteristic`` is # already snake_case-compatible (single word) and is sent unchanged. The -# remaining snake_case params are bookkeeping (``service``/``profile``/ -# ``ssl_check``) and never reach the request. +# remaining snake_case params are control arguments (``service``/``profile``/ +# ``ssl_check``) and are never sent in the request. _SAMPLES_PARAM_TO_API = { "activity_media_name": "activityMediaName", "activity_start_date_lower": "activityStartDateLower", @@ -172,9 +172,9 @@ def get_samples( """Search the USGS Samples database for discrete water-quality results. Every available filter is exposed as an argument, but leave as many as - feasible at their default of ``None``. An overcomplicated query can bog - down the database's ability to assemble a result before it times out, so - filtering narrowly is faster than filtering exhaustively. + feasible at their default of ``None``. A query with many filters can keep + the database from assembling a result before the request times out, so + a query with few filters runs faster than one with many. The web GUI for the Samples database is at https://waterdata.usgs.gov/download-samples/#dataProfile=site diff --git a/dataretrieval/waterdata/stats.py b/dataretrieval/waterdata/stats.py index 2ec7eb9e1..0087641de 100644 --- a/dataretrieval/waterdata/stats.py +++ b/dataretrieval/waterdata/stats.py @@ -2,7 +2,7 @@ Wraps ``https://api.waterdata.usgs.gov/statistics/v0`` — the daily-statistics service (period-of-record and date-range normals/intervals). This is a -*separate*, non-OGC API with no chunkable multi-value axes, so it runs as a +separate, non-OGC API with no chunkable multi-value axes, so it runs as a one-item :class:`~dataretrieval.transport.fanout.FanOut` rather than going through ``multi_value_chunked`` (ADR 0008). The typed getters ``get_stats_por`` and @@ -62,7 +62,7 @@ def _handle_nesting( ``data`` field, which is unrolled separately below via the ``record_path`` json_normalize), then adds ``geometry`` only when present. Unlike :func:`engine._get_resp_data`, no top-level ``id`` - column is added — stats features don't carry one, so this matches the + column is added — stats features do not have one, so this matches the geopandas branch. Skipping the GeoJSON envelope keeps newly-added fields like ``geometry.type`` from leaking into the result. """ @@ -95,9 +95,9 @@ def _extract_features(body: dict[str, Any] | None) -> list[dict[str, Any]] | Non """Return the features list from a response body, or None for empty/missing. ``None`` signals the caller to return an empty frame. An empty (or - missing) features list — a real mid-pagination shape — would otherwise - crash the downstream merge with ``KeyError: 'monitoring_location_id'`` - because neither frame would carry the merge key. + missing) features list — a shape that occurs mid-pagination — would otherwise + fail the downstream merge with ``KeyError: 'monitoring_location_id'`` + because neither frame would have the merge key. """ if body is None: return None @@ -117,15 +117,15 @@ def _build_outer_frame(features: list[dict[str, Any]], geopd: bool) -> pd.DataFr ] df = pd.json_normalize(outer_props, sep=".") df.columns = df.columns.str.split(".").str[-1] - # Stats features don't carry a top-level ``id`` field — the + # Stats features do not have a top-level ``id`` field — the # geopandas branch (``GeoDataFrame.from_features``) doesn't - # surface one either, so the non-geopd branch stays - # consistent by NOT adding an id column. + # add one either, so the non-geopd branch stays + # consistent by not adding an id column. _attach_coordinates(df, features) return df # Stats features may omit ``geometry`` entirely; ``_geo_feature_frame`` - # is the shared home for that upstream-schema workaround. + # applies that upstream-schema workaround for every caller. return _geo_feature_frame(features).drop(columns=["data"], errors="ignore") @@ -224,9 +224,9 @@ def get_data( computation_type other than percentiles, a percentile column is still returned. client : httpx.AsyncClient, optional - Caller-borrowed async client. ``None`` (default) borrows the one this + Caller-supplied async client. ``None`` (default) uses the one this call's :class:`~dataretrieval.transport.fanout.FanOut` opened, which - lives in the same event loop as the page walk. Primarily a test seam. + runs in the same event loop as the page walk. Primarily a test seam. Returns ------- @@ -246,7 +246,7 @@ def get_data( a hostname that does not resolve), the ``httpx`` exception chained on ``__cause__``. FanOutInterrupted - A transient failure (429 / 5xx / timeout) survived the built-in + A transient failure (429 / 5xx / timeout) was not resolved by the built-in retries. Resume with ``exc.call.resume()`` (see :doc:`/userguide/errors`). """ @@ -264,12 +264,12 @@ def get_data( def parse_response(resp: httpx.Response) -> tuple[pd.DataFrame, str | None]: body = resp.json() # Coerce falsy cursors ("", 0) to None so _paginate terminates. - # USGS uses "next": null at end-of-stream, but defensive coerce - # protects against any "" sentinel a future schema might use. + # USGS uses "next": null on the last page, but the coercion + # also covers any "" sentinel a future schema might use. return _handle_nesting(body, geopd=GEOPANDAS), body.get("next") or None async def follow_up(cursor: str, sess: httpx.AsyncClient) -> httpx.Response: - # Build a fresh params dict per page so the caller's ``args`` + # Build a new params dict per page so the caller's ``args`` # is never mutated. return await sess.request( method, url=url, params={**args, "next_token": cursor}, headers=headers diff --git a/dataretrieval/waterdata/time_series.py b/dataretrieval/waterdata/time_series.py index ee27758e1..c83232f0e 100644 --- a/dataretrieval/waterdata/time_series.py +++ b/dataretrieval/waterdata/time_series.py @@ -1,12 +1,12 @@ """Getters for observations that form a time series. Daily and continuous values, their most-recent counterparts, and the -period-of-record statistics computed over them. What unites them is shape: a +period-of-record statistics computed over them. What they share is shape: a monitoring location and a parameter, repeated over time. Metadata *about* these series -- what a location measures, over what period -- -lives in :mod:`~dataretrieval.waterdata.metadata`, so a caller can discover what -exists before asking for the observations. +is in :mod:`~dataretrieval.waterdata.metadata`, so a caller can discover what +exists before requesting the observations. """ from __future__ import annotations @@ -164,7 +164,7 @@ def get_daily( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -211,7 +211,7 @@ def get_daily( ... time="2021-01-01T00:00:00Z/2022-01-01T00:00:00Z", ... ) - >>> # Quick "show me the last week" idiom (ISO 8601 duration) + >>> # The last seven days, as an ISO 8601 duration >>> df, md = dataretrieval.waterdata.get_daily( ... monitoring_location_id="USGS-02238500", ... parameter_code="00060", @@ -226,7 +226,7 @@ def get_daily( ... ) >>> # Pull only rows whose underlying record was refreshed in the - >>> # last 7 days — handy for incremental ETL polling + >>> # last 7 days — useful for incremental ETL polling >>> df, md = dataretrieval.waterdata.get_daily( ... monitoring_location_id="USGS-02238500", ... parameter_code="00060", @@ -236,9 +236,9 @@ def get_daily( >>> # Chain queries: pull all stream monitoring locations in a >>> # state, then their daily discharge for the last week. The >>> # location list can be hundreds of values long — the request - >>> # is transparently chunked across multiple chunks so the URL - >>> # stays under the server's byte limit. Combined output looks - >>> # like a single query. + >>> # is split into several chunks so the URL stays under the + >>> # server's byte limit. The combined output is the same as for + >>> # a single request. >>> sites_df, _ = dataretrieval.waterdata.get_monitoring_locations( ... state="Ohio", ... site_type="Stream", @@ -385,7 +385,7 @@ def get_continuous( limit : int, optional The number of features returned in each page. The maximum allowable limit is 10000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -593,7 +593,7 @@ def get_latest_continuous( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -806,7 +806,7 @@ def get_latest_daily( limit : int, optional The number of features returned in each page. The maximum allowable limit is 50000; the default (None) requests that maximum. Set a lower - number if your internet connection is spotty. This is a per-page size, + number if your internet connection is unreliable. This is a per-page size, not a cap on the total result: a query matching more rows than ``limit`` still returns every matching row across multiple pages. Use ``max_rows`` to cap the total instead. @@ -898,7 +898,7 @@ def get_stats_por( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get day-of-year and month-of-year statistics over the historical record. - Answers "how does today compare to a normal day here?" -- minimum, maximum, + Compares a day to the same day of year across the record: minimum, maximum, mean, median, and percentiles computed per day of year and month of year (the ``observationNormals`` endpoint). For more on how these statistics are calculated, see the Statistics documentation page: @@ -943,7 +943,7 @@ def get_stats_por( The number of results to return per page, where one result represents a monitoring location. The default is 1000. parent_time_series_id: string, optional - Returns statistics tied to a particular database entry. + Returns statistics associated with a particular database entry. site_type_code: string, optional Site type code query parameter. A list of valid site type codes is available at @@ -1038,7 +1038,7 @@ def get_stats_date_range( ) -> tuple[pd.DataFrame, BaseMetadata]: """Get statistics summarizing whole months and years of the record. - Answers "how did this month or year compare to others?" -- minimum, maximum, + Compares a month or year to the others in the record: minimum, maximum, mean, median, and percentiles per month-year and per water or calendar year (the ``observationIntervals`` endpoint). For more on how these statistics are calculated, see the Statistics documentation page: @@ -1085,7 +1085,7 @@ def get_stats_date_range( The number of results to return per page, where one result represents a monitoring location. The default is 1000. parent_time_series_id: string, optional - Returns statistics tied to a particular database entry. + Returns statistics associated with a particular database entry. site_type_code: string, optional Site type code query parameter. A list of valid site type codes is available at diff --git a/dataretrieval/waterdata/types.py b/dataretrieval/waterdata/types.py index c234513df..bbc7dd453 100644 --- a/dataretrieval/waterdata/types.py +++ b/dataretrieval/waterdata/types.py @@ -54,10 +54,10 @@ "results", ] -# OGC API time-series/monitoring collections queryable via ``get_cql``. -# Keep in sync with ``utils._OUTPUT_ID_BY_COLLECTION`` (same keys): that dict maps -# each service to its user-facing ``id`` column and is the runtime source of -# truth ``get_cql`` validates against. +# OGC API time-series/monitoring collections queryable via ``get_cql``. Keep in sync +# with ``utils._OUTPUT_ID_BY_COLLECTION`` (same keys): that dict maps each service to +# its user-facing ``id`` column and is the runtime definition ``get_cql`` validates +# against. WATERDATA_COLLECTIONS = Literal[ "channel-measurements", "combined-metadata", diff --git a/dataretrieval/waterdata/utils.py b/dataretrieval/waterdata/utils.py index cfdf10c65..ede2e1171 100644 --- a/dataretrieval/waterdata/utils.py +++ b/dataretrieval/waterdata/utils.py @@ -3,11 +3,11 @@ This module is the Water-Data-specific adapter: it supplies the collection-to-id map, the CQL2/date-only dialect, and a thin ``get_ogc_data`` wrapper that injects the Water Data defaults. The -statistics path lives in its own :mod:`dataretrieval.waterdata.stats` +statistics path is in its own :mod:`dataretrieval.waterdata.stats` module. -OGC machinery (request construction, pagination, response shaping, the -chunked ``get_ogc_data`` entry point) lives in :mod:`dataretrieval.ogc` +OGC protocol code (request construction, pagination, response shaping, the +chunked ``get_ogc_data`` entry point) is in :mod:`dataretrieval.ogc` and its implementation submodules. This adapter consumes the public facade for dialects, argument normalization, and retrieval; it is not a re-export layer for OGC helpers (ADR 0003). @@ -62,11 +62,11 @@ "time-series-metadata": "time_series_id", } -# Every collection's output id EXCEPT the two that are genuinely user-facing +# Every collection's output id except the two that are user-facing # (``monitoring_location_id`` and ``time_series_id``). The rest are synthetic # per-record ids that ``_arrange_cols`` moves to the end of a result frame. -# Derived from ``_OUTPUT_ID_BY_COLLECTION`` so adding a collection can't silently -# leave a stray id column at the front again. +# Derived from ``_OUTPUT_ID_BY_COLLECTION`` so adding a collection cannot +# leave an extra id column at the front again. _EXTRA_ID_COLS = frozenset( set(_OUTPUT_ID_BY_COLLECTION.values()) - {"monitoring_location_id", "time_series_id"} @@ -106,9 +106,9 @@ sort_cols=("time", "monitoring_location_id"), ) -# The Water-Data-specific *extras* on top of the engine's own no-normalize set +# The Water-Data-specific extras in addition to the engine's own no-normalize set # (which already covers the date-range params and ``bbox``). Scalar non-string -# knobs are caught by runtime type, so only iterables with special handling +# parameters are caught by runtime type, so only iterables with special handling # need to be named here: # - ``boundingBox`` is ``list[float]``, sometimes ``numpy.ndarray`` # - ``get_peaks``'s int-valued filters (``water_year`` etc.) are ``list[int]`` @@ -129,20 +129,20 @@ def _flatten_queryables(local_vars: dict[str, Any]) -> dict[str, Any]: """Merge a getter's ``**queryables`` passthrough kwargs into ``local_vars``. - ``locals()`` collects them under the ``queryables`` key; this lifts them to + ``locals()`` collects them under the ``queryables`` key; this moves them to top-level entries, so an extra server-side filter such as ``state_name="Wisconsin"`` is normalized, mutual-exclusion-checked, and sent exactly like a named param. See :func:`dataretrieval.waterdata.get_queryables` for each collection's filterable properties (the collection rejects an unknown one with a 400). - ``**queryables`` always arrives as a dict (empty when unused) and the key is + ``**queryables`` is always passed as a dict (empty when unused) and the key is popped, so this is a no-op on getters without the passthrough and idempotent if called twice. """ queryables = local_vars.pop("queryables", {}) - # A credential-shaped name would go out in the query string, which is the - # one thing this passthrough must not forward. The predicate lives in the + # A credential-shaped name would be sent in the query string, which is the + # one thing this passthrough must not forward. The predicate is defined in the # credentials leaf rather than here: what motivates it -- ``api_key=`` being # a plausible guess now that ``configure()`` takes it -- is package-wide, # and WQP's ``**kwargs`` search filters read the same list. @@ -178,11 +178,11 @@ def _with_state(local_vars: dict[str, Any], *, to: str, into: str) -> dict[str, ``state_name`` parameters, which still accept the API's raw values (e.g. non-US FIPS); passing ``state`` together with either raises ``ValueError``. """ - # Flatten ``**queryables`` first so a native state param arriving that way + # Flatten ``**queryables`` first so a native state param passed that way # (e.g. ``get_time_series_metadata``'s ``state_code``, which isn't an - # explicit parameter) is visible to apply_state's mutual-exclusion guard. - # Otherwise ``state`` plus a passthrough ``state_code`` would slip past the - # check and silently send both. + # explicit parameter) is visible to apply_state's mutual-exclusion check. + # Otherwise ``state`` plus a passthrough ``state_code`` would pass the + # check and send both. _flatten_queryables(local_vars) return apply_state( local_vars, to=to, into=into, reject=("state_code", "state_name") @@ -224,7 +224,7 @@ def get_ogc_data( A verbatim CQL2 JSON body to POST instead of building the query from ``args`` (see the facade's ``cql_body``). Used by :func:`get_cql`. spatial : bool, optional - Whether the collection carries feature geometry. Water Data's typed + Whether the collection includes feature geometry. Water Data's typed feature collections do; reference tables pass ``False``. Returns @@ -242,7 +242,7 @@ def get_ogc_data( collection, output_id, max_rows=max_rows, - # Endpoint acquisition resolves the active ContextVar at request time; + # The endpoint is resolved from the active ContextVar at request time; # the documented ``OGC_API_URL`` constant remains the default-value # compatibility path rather than a production request destination. base_url=ogc_api_url(), @@ -253,7 +253,7 @@ def get_ogc_data( # Which settings table these calls read. Declared here, in the one # wrapper every Water Data getter goes through, rather than derived # from ``base_url``: NGWMN is served from the same host, so a URL - # cannot tell the two adapters apart (ADR 0010). + # does not distinguish the two adapters (ADR 0010). adapter="waterdata", ) @@ -284,19 +284,18 @@ def _accept_legacy_kwargs( so static checkers won't flag legacy call sites. ``removal`` is the published horizon (from - :data:`~dataretrieval._deprecation.REMOVALS`); ``None`` reads as "a future - release". ``detail`` appends a sentence to the warning. The default - message says only - that the name changed; a rename with a reason worth giving -- a spec that - names the value differently, a removal date -- passes it here rather than - hand-rolling the whole shim to carry one sentence. + :data:`~dataretrieval._deprecation.REMOVALS`); ``None`` is rendered as "a future + release". ``detail`` appends a sentence to the warning. The default message says + only that the name changed; a rename with a reason to state -- a spec that names the + value differently, a removal date -- passes it here rather than writing a whole shim + to hold one sentence. Raises ------ TypeError - If both a deprecated name and its modern equivalent are supplied for - the same argument (ambiguous), mirroring Python's "got multiple - values for argument" error. + If both a deprecated name and its modern equivalent are supplied for the same + argument (ambiguous), matching Python's "got multiple values for argument" + error. """ def decorator(func: Callable[..., _R]) -> Callable[..., _R]: diff --git a/dataretrieval/wateruse.py b/dataretrieval/wateruse.py index 35346a61b..3580607d1 100644 --- a/dataretrieval/wateruse.py +++ b/dataretrieval/wateruse.py @@ -13,18 +13,18 @@ -- so calls and identity comparisons behave identically through either spelling. -It is an alias for reading, not a second name for the module. This is a +It aliases the public names; it is not a second name for the module object. This is a distinct module object holding its own references to the five public names, so it does not forward *assignment* or private names: rebinding -``wateruse.get_wateruse`` leaves ``nwdc``'s global untouched (and so has no +``wateruse.get_wateruse`` leaves ``nwdc``'s global unchanged (and so has no effect on anything ``nwdc`` does internally), and ``wateruse._WATERUSE_HOST`` -does not exist. Code that monkeypatches, or that reaches for a private, must +does not exist. Code that monkeypatches, or that uses a private name, must name :mod:`dataretrieval.nwdc` directly -- which is the point of the deprecation. ``dataretrieval.__init__`` deliberately imports :mod:`dataretrieval.nwdc` -rather than this module, so ``import dataretrieval`` stays silent. The warning -fires only for code that names ``wateruse`` itself. +rather than this module, so ``import dataretrieval`` emits no warning. The warning +is emitted only for code that names ``wateruse`` itself. """ from __future__ import annotations @@ -33,10 +33,9 @@ from dataretrieval._deprecation import REMOVALS, warn_deprecated from dataretrieval.nwdc import * # noqa: F403 (re-export the public surface) -#: When the alias may be deleted. Read from the shared horizon table rather -#: than spelled here, so it is audited and bumped with every other published -#: removal; matches the dated-removal convention :mod:`dataretrieval.nwis` -#: uses. +#: When the alias may be deleted. Read from the shared horizon table rather than spelled +#: here, so it is reviewed and extended with every other published removal; matches the +#: dated-removal convention :mod:`dataretrieval.nwis` uses. NWDC_RENAME_REMOVAL_DATE = REMOVALS["wateruse"] __all__ = list(_nwdc.__all__) @@ -47,7 +46,7 @@ removal=NWDC_RENAME_REMOVAL_DATE, detail="The service is the National Water Availability Assessment Data " "Companion, and water use is one of the ten datasets it serves.", - # 1 lands the warning on the line that imported this module -- an import - # has no deeper user frame to point at. + # 1 attributes the warning to the line that imported this module -- an import + # has no deeper user frame to attribute it to. stacklevel=1, ) diff --git a/dataretrieval/wqp.py b/dataretrieval/wqp.py index 0c135ea97..8538ffd1c 100644 --- a/dataretrieval/wqp.py +++ b/dataretrieval/wqp.py @@ -56,7 +56,7 @@ #: Root the Water Quality Portal serves both its interfaces from. Private #: because the two builders below are the documented way to name a WQP URL; -#: this is only the piece they share, and the piece a redirect replaces. +#: this is only the part they share, and the part a redirect replaces. _WQP_BASE_URL = "https://www.waterqualitydata.us" result_profiles_wqx3 = ["basicPhysChem", "fullPhysChem", "narrow"] @@ -137,11 +137,11 @@ def _read_wqp_csv(text: str) -> DataFrame: """Read a WQP CSV, forcing code/identifier columns to ``str``. WQP returns codes with significant leading zeros — HUCs, parameter codes - (``USGSpcode``), FIPS state/county codes. A bare ``read_csv`` infers those - as int/float and silently drops the zeros (``"00060"`` -> ``60``, HUC8 + (``USGSpcode``), FIPS state/county codes. A bare ``read_csv`` infers those as + int/float and drops the zeros without a warning (``"00060"`` -> ``60``, HUC8 ``"07090002"`` -> ``7090002``). Read the header first, then re-read with - ``dtype=str`` for every column that :func:`_is_code_column` flags, so the - zeros survive. + ``dtype=str`` for every column that :func:`_is_code_column` flags, so the zeros are + preserved. """ columns = pd.read_csv(StringIO(text), delimiter=",", nrows=0).columns str_cols = {col: str for col in columns if _is_code_column(col)} @@ -680,7 +680,7 @@ def _service_base() -> str: The portal serves the legacy and WQX3 interfaces from one root under different paths, so a ``WqpConfiguration(base_url=...)`` names that root and - both follow it (ADR 0011). + both use it (ADR 0011). """ return _configuration.base_url(adapter="wqp", default=_WQP_BASE_URL) @@ -759,11 +759,11 @@ def site_info(self) -> tuple[DataFrame, WQP_Metadata] | None: def _check_kwargs(kwargs: dict[str, Any]) -> dict[str, Any]: """Check kwargs for unsupported parameters. - Every WQP getter's ``**kwargs`` funnels through here on its way to the - query payload, so this is the choke point where a credential-shaped name is + Every WQP getter's ``**kwargs`` passes through here before it becomes the + query payload, so this is the one place where a credential-shaped name is refused. The predicate is the credentials leaf's, shared with Water Data's ``**queryables`` passthrough: ``api_key=`` is a plausible guess on any - getter now that ``configure(Configuration(api_key=...))`` is the spelling, + getter now that ``configure(Configuration(api_key=...))`` is the documented form, and this is the adapter with the widest passthrough -- ten getters, whose filter names the portal rather than this package defines. The returned payload materializes non-string iterables as lists so one-shot iterators @@ -827,8 +827,8 @@ def _legacy_only_url(service: str, legacy: bool) -> str: Passing ``legacy=False`` to one of these helpers emits a ``UserWarning`` explaining the fallback and *also* suppresses the legacy :class:`~dataretrieval.exceptions.DataCurrencyWarning` that ``wqp_url`` - would otherwise raise. That warning's message claims setting - ``legacy=False`` removes it, which is a lie for endpoints that have no + would otherwise raise. That warning's message states that setting + ``legacy=False`` removes it, which is false for endpoints that have no WQX3.0 alternative. """ with warnings.catch_warnings(): @@ -842,8 +842,8 @@ def _legacy_only_url(service: str, legacy: bool) -> str: class WqpConfiguration(_Redirectable, _Retrying, BaseConfiguration): """Settings for Water Quality Portal calls alone. - No fan-out dials: a WQP query is answered by a single request, so a - concurrency cap could only report a number nothing honours. + No fan-out settings: a WQP query is served by a single request, so a + concurrency cap could only report a number nothing reads. Declared here rather than in :mod:`dataretrieval.configuration` (ADR 0011). @@ -857,7 +857,7 @@ class WqpConfiguration(_Redirectable, _Retrying, BaseConfiguration): stops. base_url : str, optional Root to send WQP requests to, instead of the portal's own. Both - interfaces hang off it, so one value moves the legacy ``/data/`` + interfaces are built on it, so one value redirects the legacy ``/data/`` and the WQX3 ``/wqx3/`` paths together. Code only: the file and the environment refuse it. """ diff --git a/demos/nwqn_data_pull/README.md b/demos/nwqn_data_pull/README.md index c3f6171e1..8f2b6e426 100644 --- a/demos/nwqn_data_pull/README.md +++ b/demos/nwqn_data_pull/README.md @@ -19,7 +19,7 @@ pip install -r requirements.txt 2. Configure compute and storage backends for [lithops](https://lithops-cloud.github.io/docs/source/configuration.html). The configuration in `lithops.yaml` uses AWS Lambda for [compute](https://lithops-cloud.github.io/docs/source/compute_config/aws_lambda.html) and AWS S3 for [storage](https://lithops-cloud.github.io/docs/source/storage_config/aws_s3.html). -To use those backends, simply edit `lithops.yaml` with your `bucket` and `execution_role`. +To use those backends, edit `lithops.yaml` with your `bucket` and `execution_role`. 3. Build a runtime image for Cubed ```bash diff --git a/demos/nwqn_data_pull/retrieve_nwqn_streamflow.py b/demos/nwqn_data_pull/retrieve_nwqn_streamflow.py index fc6a5cfbc..f5ebaf498 100644 --- a/demos/nwqn_data_pull/retrieve_nwqn_streamflow.py +++ b/demos/nwqn_data_pull/retrieve_nwqn_streamflow.py @@ -43,7 +43,7 @@ def map_retrieval(site): print(len(df), "records retrieved") # process the results if not df.empty: - # drop rows with missing values; neglect other 00060_* columns + # drop rows with missing values; ignore other 00060_* columns df = df.dropna(subset=["00060_Mean"]) # fill missing codes to enable string operations df["00060_Mean_cd"] = df["00060_Mean_cd"].fillna("M") diff --git a/docs/source/architecture/decisions/0000-documenting-decisions.rst b/docs/source/architecture/decisions/0000-documenting-decisions.rst index 4ae3da8b5..253504606 100644 --- a/docs/source/architecture/decisions/0000-documenting-decisions.rst +++ b/docs/source/architecture/decisions/0000-documenting-decisions.rst @@ -12,7 +12,7 @@ section records the clause added. Context ------- -This package is documented heavily and deliberately. Its public getters are +This package is documented extensively and deliberately. Its public getters are thin wrappers whose numpydoc parameter tables *are* the deliverable: 55% of all docstring lines in ``dataretrieval/`` are in the service adapters, at a ratio of 2.5 prose lines per line of code. CONTRIBUTING already requires those tables. @@ -20,7 +20,7 @@ docstring lines in ``dataretrieval/`` are in the service adapters, at a ratio of The problem is in the internal modules behind them. Rationale -- the argument for why a rule holds -- accumulated in module and function docstrings alongside the ADRs that already owned it, because a paragraph can be written where the -reader already is, while a citation sends them to a record they have to open. +reader already is, while a citation requires them to open a record. Those modules hold 82% of the package's comment lines, and an audit of that prose found roughly 500 lines restating decisions already recorded in ADRs 0003 through 0011: ``configuration.py`` re-derives the layered-resolution design in @@ -29,7 +29,7 @@ the no-progress budget is argued from first principles in five places across ``transport/``, and which failures may be retried is enumerated in three lists that can drift apart. -Duplication is not a tidiness problem here; it is a correctness problem. Every +Duplication here is a correctness problem. Every copy is a place the rule can be updated while the others are not, and the audit found copies that had already gone stale -- an overview paragraph describing concurrency caps that a later ADR had removed, and an ADR clause describing a @@ -57,7 +57,7 @@ than the lines beneath it, and belongs in one of the venues below. **Commit messages own the history.** Benchmark numbers, the symptom that prompted a change, what the code used to do, what was tried and rejected. This is the venue with a date and a diff attached. It is the one place where "was -once optional" or "measured 1.6x slower" stays true forever without maintenance. +once optional" or "measured 1.6x slower" stays true without maintenance. Source files describe the current state, not how it was reached. **ADRs own the cross-cutting decision.** A choice that constrains code outside @@ -68,7 +68,7 @@ number a new one sequentially and follow :doc:`template`. **The glossary owns the vocabulary.** ``CONTEXT.md`` defines terms with package-wide meaning. Documents use those terms rather than redefining them, and -where a term and the code disagree, the term is authoritative. +where a term and the code differ, the term is authoritative. Three rules follow: @@ -94,7 +94,7 @@ Consequences opening an ADR. That cost is accepted -- the reader who needs the argument is rarer than the reader who needs the contract, and the ADR is the version that is maintained. -- Rationale is not deleted when it moves. Prose that leaves a docstring is moved +- Rationale is not deleted when it moves. Prose removed from a docstring is moved to an ADR clause or to the commit message that removes it. The commit message is where a reviewer looks for what a documentation change discarded. - Docstring volume in the service adapters is expected to stay high and is not a @@ -102,7 +102,7 @@ Consequences whether it is over-documented. - The policy applies going forward. Existing prose is migrated when a module is being changed for another reason, rather than in a single pass that would - touch every file at once. + change every file at once. Compliance ---------- @@ -110,7 +110,7 @@ Compliance Reviewers apply two questions to added prose. First: *does this explain the lines beneath it, or does it argue for a rule that binds another file?* The second belongs in an ADR, cited by number. Then: *could a reader who has not -opened the cited record follow this sentence?* If not, the citation has hidden +opened the cited record follow this sentence?* If not, the citation has removed the explanation rather than relocated it. The repair is to give the reader what they need -- name the term, resolve the pronoun, say which venue owns the rest -- not to restate the argument the citation replaced. @@ -122,8 +122,8 @@ docstring or comment that names an ADR must name one that exists. ``docs/source/architecture/decisions/``, so a renumbered or deleted record fails the suite rather than leaving a dangling pointer. Whether a given paragraph should have been a citation remains a review judgement. No test is proposed: a -proxy metric here would push contributors to delete parameter documentation to -improve a number. +proxy metric here would encourage contributors to delete parameter +documentation to improve a number. Notes ----- @@ -134,7 +134,7 @@ this record had put in the right venue and that only the author could follow: undefined jargon, a pronoun with no antecedent, and a mapping between two numbering schemes that needed a second document open. One instance broke this record's own history rule. The venue rules say where an explanation -goes; none of them considered who would read it. +goes; none of them addressed who would read it. ``Context`` and the measurements below are this package's. ``Decision``, ``Consequences``, and the review questions in ``Compliance`` are written to hold diff --git a/docs/source/architecture/decisions/0001-modular-monolith.rst b/docs/source/architecture/decisions/0001-modular-monolith.rst index 68b22e7ce..befcc0d6a 100644 --- a/docs/source/architecture/decisions/0001-modular-monolith.rst +++ b/docs/source/architecture/decisions/0001-modular-monolith.rst @@ -22,11 +22,11 @@ Decision Maintain one installable distribution organized as a modular monolith. Expose functions grouped by data portal. Keep service- and protocol-specific adapters independent behind those facades, and share infrastructure only where its -contract is genuinely API-neutral. +contract is API-neutral. Treat the OGC subsystem as a protocol component used by Water Data and NGWMN, not as a universal service framework. Do not force NLDI, StreamStats, WQP, or -Water Use into OGC-shaped return values or paging semantics. +Water Use into return values or paging semantics in OGC's form. Consequences ------------ diff --git a/docs/source/architecture/decisions/0003-dependency-direction.rst b/docs/source/architecture/decisions/0003-dependency-direction.rst index 41eb9090f..434ba9fbb 100644 --- a/docs/source/architecture/decisions/0003-dependency-direction.rst +++ b/docs/source/architecture/decisions/0003-dependency-direction.rst @@ -22,7 +22,7 @@ Dependencies point from public facades to service/protocol adapters, then to service-neutral transport and stable policy, and finally to third-party infrastructure. In particular: -- ``dataretrieval.exceptions`` is a runtime-dependency-light leaf. +- ``dataretrieval.exceptions`` is a leaf with no runtime third-party dependencies. - ``dataretrieval.ogc`` must not import Water Data, NGWMN, Water Use, or NWIS. - ``dataretrieval.ogc`` must not depend on the mixed legacy ``utils`` module; shared scoped state is kept in a dependency-free leaf instead. @@ -65,14 +65,14 @@ direction belongs in ``.importlinter``. Named contracts verify the current boundaries: the only OGC dependency of NGWMN and ``waterdata.cql`` is the facade, ``ogc.shaping`` does not depend on -``ogc.engine``, Water Use and the other non-OGC adapters cannot reach the OGC +``ogc.engine``, Water Use and the other non-OGC adapters cannot import the OGC subsystem at all. The fitness functions verify that the runtime graph is acyclic package-wide rather than only within ``ogc`` and ``transport``. ``waterdata.utils`` not bulk re-exporting private OGC helpers stays there too, because that claim is about the module's ``__all__``. -The OGC consumer list is an allowlist, so a new service module is refused until +The OGC consumer list is an allowlist, so a new service module fails the contract until someone places it deliberately. It should shrink as private seams move. Any growth requires explicit architecture review, and a change to the dependency policy requires this ADR to be superseded. diff --git a/docs/source/architecture/decisions/0004-error-retry-resume.rst b/docs/source/architecture/decisions/0004-error-retry-resume.rst index 466bb2524..88afda88a 100644 --- a/docs/source/architecture/decisions/0004-error-retry-resume.rst +++ b/docs/source/architecture/decisions/0004-error-retry-resume.rst @@ -43,8 +43,8 @@ covers what stops a call; two things that do not stop one are decided here as well, because getting either wrong turns a condition that should not stop a call into one that does. -A fan-out over *independent* items may skip one. Where a query asks for many -items that do not compose into a single answer, an item failing +A fan-out over *independent* items may skip one. Where a query requests many +items that do not compose into a single result, an item failing deterministically is dropped with a warning naming it, and counts as complete so a resume does not re-attempt it. A transient failure is never skipped: it retries, and once retries are exhausted it raises a resumable interruption like @@ -73,14 +73,14 @@ rather than ``cls(*args)``, because these errors have fields whose values are not the constructor's arguments. A subclass holding an unpicklable handle -- a client, a task -- must remove it in ``__getstate__``. Without this a failure raised inside a worker process is replaced by a pickling error as it is -returned, losing the diagnosis exactly when it is hardest to reproduce. +returned, losing the diagnosis when it is hardest to reproduce. Consequences ------------ - Callers can catch one stable base error and still branch on ``status_code``, ``retry_after``, and ``retryable``. -- Mid-pagination failure cannot silently look like a complete dataset. +- Mid-pagination failure cannot be mistaken for a complete dataset. - Retry can increase latency and request quota, so policy and defaults are part of observable behavior. - Partial OGC state requires serialization and finalization tests. @@ -95,7 +95,7 @@ pagination failure, retry exhaustion and jitter bounds, ``Retry-After`` limits, resume equivalence, partial-state stability, pickling, and cancellation precedence. The skip policy is covered by ``tests/waterdata_ratings_test.py::test_get_ratings_deterministic_download_failure_warns_and_skips`` -and its sibling for a feature with no asset; the warning categories by +and the companion test for a feature with no asset; the warning categories by ``tests/deprecation_test.py``, which asserts ``DataCurrencyWarning`` is not a subclass of ``DeprecationWarning``; the ``Retry-After`` parsing rules by the ``Retry-After`` date and over-cap cases; and the process boundary by @@ -108,7 +108,7 @@ The warning, ``Retry-After`` parsing, and pickling clauses were added after the original decision. They record, under ADR 0000, rules the code was stating in prose. The skip clause records an exception that previously read as contradicting this record and :doc:`0006-service-neutral-transport`; 0006 now -points here for it. +cites this record for it. The ``Status`` line was also annotated retroactively: this record assigned resumable partial state to OGC, which :doc:`0008-fan-out-execution` superseded diff --git a/docs/source/architecture/decisions/0005-legacy-nwis.rst b/docs/source/architecture/decisions/0005-legacy-nwis.rst index 5d9be3b5e..a1b8771b6 100644 --- a/docs/source/architecture/decisions/0005-legacy-nwis.rst +++ b/docs/source/architecture/decisions/0005-legacy-nwis.rst @@ -36,7 +36,7 @@ Consequences - Maintainers avoid investing in a second implementation of modern retrieval behavior. - Legacy integration tests may require special handling as upstream endpoints - disappear. + are retired. - Removal still requires release notes, replacement checks, and an intentional compatibility boundary. diff --git a/docs/source/architecture/decisions/0006-service-neutral-transport.rst b/docs/source/architecture/decisions/0006-service-neutral-transport.rst index f5cb1ce97..dec857a6d 100644 --- a/docs/source/architecture/decisions/0006-service-neutral-transport.rst +++ b/docs/source/architecture/decisions/0006-service-neutral-transport.rst @@ -75,7 +75,7 @@ upstream failure may have ended, while a single-shot adapter re-sends only the gateway statuses, because its service responds to a *rejected query* with a 500, and re-sending that would spend a caller's quota on a request that cannot succeed. Both sets are narrower than ``DataRetrievalError.retryable``, -deliberately: that field tells a caller re-issuing might work, where spending +deliberately: that field indicates re-issuing might work, where spending someone's quota unasked needs a stricter criterion. Deprecated NWIS calls retain their compatibility behavior. A failed pagination or fan-out operation raises rather than returning successful siblings as an apparently complete @@ -85,18 +85,18 @@ recorded in :doc:`0004-error-retry-resume`. Two independent bounds limit retry: an attempt count and a no-progress budget measured in seconds since data last arrived. Attempts alone leave elapsed time unbounded, since each attempt may itself block until its timeout; the budget -alone would cut short a slow but productive download. Receiving a page restarts +alone would stop a slow but productive download early. Receiving a page restarts the budget, and an attempt already in flight is never interrupted. **Time spent waiting is not time without progress.** The budget bounds time -the *service* left the caller with nothing, so time the package chose to spend +the *service* left the caller with nothing, so time the package itself spent waiting is excluded by the measured amount: a wait the server named in ``Retry-After``, and time a chunk spent waiting for the concurrency semaphore. The first retry is exempt outright. Without these exemptions a policy that follows a server's ``Retry-After`` would spend its own budget doing so, and a call would lose retries for being throttled by settings the caller chose. An exclusion never sets the reference time later than now: a timestamp ahead of -now would make the elapsed no-progress time negative and silently disable the +now would make the elapsed no-progress time negative and disable the bound. Because half of that exclusion is the retry driver's, the concurrency semaphore is acquired *per attempt* inside the retry driver rather than held by the caller across one. @@ -128,8 +128,8 @@ Consequences - Retry can increase latency and quota consumption, so attempt counts, waits, and total no-progress time remain bounded, and cancellation signals are never wrapped. -- Guidance the progress reporter prints depends on the host it applies to, so a - service that cannot use an API key is not told to obtain one. +- Guidance the progress reporter prints depends on the host it applies to, so the + advice to obtain one is not printed for a service that cannot use it. - The transport package is internal infrastructure, not a new public API contract. - Keeping presentation and frame assembly out means transport is roughly 570 diff --git a/docs/source/architecture/decisions/0007-adapter-facades.rst b/docs/source/architecture/decisions/0007-adapter-facades.rst index a8b35aa6e..a00fa89e6 100644 --- a/docs/source/architecture/decisions/0007-adapter-facades.rst +++ b/docs/source/architecture/decisions/0007-adapter-facades.rst @@ -43,7 +43,7 @@ exports. Deprecated NWIS remains outside this modernization. Service adapters do not import another adapter's implementation to obtain transport behavior. The ``__module__`` rule above is scoped to this facade, where the family module -is a real file a traceback can name. It is not a package-wide prohibition: the +is a file that exists for a traceback to name. It is not a package-wide prohibition: the legacy ``dataretrieval.utils`` names are split across private modules by dependency and *do* report the documented path, because there the alternative is a public, documented import location pointing at a private module. @@ -62,7 +62,7 @@ package-wide, legacy NWIS included: it is about what an adapter returns, not how it is organized. HUCs, parameter codes, FIPS codes, and monitoring-location identifiers (``site_no`` in NWIS) have significant leading zeros, and a bare ``read_csv`` infers them as integers and -drops those zeros -- ``"00060"`` becomes ``60``, so the value is silently wrong +drops those zeros -- ``"00060"`` becomes ``60``, so the value is wrong, with no indication, rather than missing. Every adapter reading a USGS tabular response names its identifier columns as ``str`` before parsing, which is why a two-pass header read is not a redundancy to be optimized away. @@ -87,7 +87,7 @@ Consequences Compliance ---------- -``tests/contracts/public_api_test.py`` freezes Water Data imports, signatures, +``tests/contracts/public_api_test.py`` records Water Data imports, signatures, facade identity, and compatibility names. ``tests/architecture_test.py`` requires a logic-free facade, exact active-service exports, and separate OGC request construction and schema execution. ``.importlinter`` keeps the diff --git a/docs/source/architecture/decisions/0008-fan-out-execution.rst b/docs/source/architecture/decisions/0008-fan-out-execution.rst index f29fee173..b174f511e 100644 --- a/docs/source/architecture/decisions/0008-fan-out-execution.rst +++ b/docs/source/architecture/decisions/0008-fan-out-execution.rst @@ -17,8 +17,8 @@ Context Two services turn one logical query into several requests, for unrelated reasons. A Water Data or NGWMN query whose URL exceeds the server's byte limit is split along its multi-value axes. A Water Use query naming several locations -is split because the NWDC accepts one ``location=`` per request -- its URLs run -around 63 bytes against an 8000-byte budget, so the byte limit has nothing to do +is split because the NWDC accepts one ``location=`` per request -- its URLs are +about 63 bytes against an 8000-byte budget, so the byte limit has nothing to do with it. Chunking is how you divide the data; fan-out is how you distribute the work. @@ -33,15 +33,15 @@ correct while a byte plan was the only thing anyone fanned out over. It stopped being correct once Water Use fanned out too: unable to import an OGC-internal executor, ``wateruse._fan_out`` re-implemented the semaphore, the ``asyncio.gather``, and the cancellation-before-HTTP-error failure precedence, -with a comment naming ``ChunkedCall._run`` as the original. One rule, two -copies, kept in agreement by that comment. +with a comment naming ``ChunkedCall._run`` as the original. One rule existed in +two copies, kept in agreement only by that comment. -The duplicate was not merely redundant. It lacked resume, so a rate limit +The duplicate was also defective. It lacked resume, so a rate limit partway through discarded every location that had already succeeded -- against an hourly quota, on fan-outs of hundreds of locations. It reported no progress. And it read its own module-global concurrency cap, so a user setting -``API_USGS_CONCURRENT`` to lower the request rate found one adapter ignoring -them. +``API_USGS_CONCURRENT`` to lower the request rate found that one adapter did not +apply it. Decision -------- @@ -88,7 +88,7 @@ test would have to assert they agree. Every adapter whose chunks are already a list would also need a wrapper class whose only purpose is renaming ``len``. -With the standard names a plain ``list`` is a plan, which is exactly what Water +With the standard names a plain ``list`` is a plan, which is what Water Use passes. ``ChunkPlan`` keeps ``total`` and ``iter_chunk_args`` as its own vocabulary and defines the dunders to delegate to them, so the two cannot disagree. @@ -139,9 +139,9 @@ Consequences connection failure now raises ``ServiceInterrupted`` / ``QuotaExhausted`` rather than ``ServiceUnavailable`` / ``RateLimited`` / ``NetworkError``. All remain ``DataRetrievalError``, so broad handlers are unaffected, but a narrow - handler around a Water Use call must widen. This is convergence, not novelty - -- it is what the OGC getters have always done -- and it is what makes the - failure resumable. Deterministic connection failures remain ``NetworkError``. + handler around a Water Use call must widen. The OGC getters have always + raised these types, and raising them is what makes the failure resumable. + Deterministic connection failures remain ``NetworkError``. - **Breaking:** ``wateruse.MAX_CONCURRENT_REQUESTS`` is removed in favor of ``API_USGS_CONCURRENT`` and ``wateruse.DEFAULT_CONCURRENT_REQUESTS``. - Resume re-issues a failed location's entire page walk, so pages fetched before diff --git a/docs/source/architecture/decisions/0009-layered-configuration.rst b/docs/source/architecture/decisions/0009-layered-configuration.rst index 3dc4bad56..3d22ab0c6 100644 --- a/docs/source/architecture/decisions/0009-layered-configuration.rst +++ b/docs/source/architecture/decisions/0009-layered-configuration.rst @@ -24,7 +24,7 @@ every service accepts the same settings -- is false. - **The refusal of a configuration object**, stated in the leaf clause ("a scoped action, not a ``Configuration`` dataclass") and in "A configuration object would have no way to reach the call". ``configure()`` now takes - exactly such objects. The grounds were that an instance had no way to reach a + exactly such objects. The grounds were that an instance had no way to be passed to a free function; the ``ContextVar`` this ADR established is one, and ADR 0010 had already narrowed the objection to a payload-shape preference. @@ -76,7 +76,7 @@ Supporting decisions: *blank* environment variable does not count as set, so it cannot shadow the file: container and CI tooling routinely creates one. The exception is ``progress``, where a blank ``API_USGS_PROGRESS`` has always meant "off" -- - so "does blank count as a value?" is a property of the setting + so whether blank counts as a value is a property of the setting (``configuration._BLANK_MEANS_SET``) rather than an extra tier in the chain. - **The environment ranks above the file.** This follows the precedence used by `pip @@ -103,15 +103,15 @@ Supporting decisions: value, remain compatible without making the new surfaces equally permissive. - **Each setting's policy is a row in a named table, never a branch in shared code.** Type, bounds, and parser are declared as data, guarded at import time - for completeness, so adding a setting cannot silently inherit whatever the + for completeness, so adding a setting cannot inherit, with no error, whatever the fallback branch happened to do. The rejected alternative -- an ``if``/``elif`` - chain with an implicit integer default -- fails by omission, and fails - quietly. + chain with an implicit integer default -- fails by omission, without an + error. - **The file format is forward compatible; the table layout is not.** A key the running version does not recognize warns and is ignored, so a file written for a newer release still loads rather than breaking a caller who downgraded. A key the version *does* recognize, placed in a table that cannot use it, raises: - that is a mistake the caller can fix, and ignoring it silently would leave the + that is a mistake the caller can fix, and ignoring it would leave the user believing a setting is in effect when it is not. - **Credential-shaped keyword refusal is a usability check, not a security control.** Names are matched as substrings after separators are stripped, and @@ -170,7 +170,7 @@ Supporting decisions: - **One flat set of setting names, shared by every service.** ``concurrency`` means the same thing to every adapter, so the chain resolves one name rather - than one per service. Services differ in the *value* they want, not the + than one per service. Services differ in the *value* they use, not the vocabulary, and that difference is expressed as a caller-supplied default: ``wateruse`` passes its ``DEFAULT_CONCURRENT_REQUESTS`` of 4 to ``configuration.concurrency()`` where the OGC getters take the package default @@ -190,7 +190,7 @@ Supporting decisions: matrix rather than a list, and that cost should be paid only when a requirement exists. -- **A configuration object would have no way to reach the call.** The public +- **A configuration object would have no way to be passed to the call.** The public surface is free functions -- ``waterdata.get_daily(...)``, not a client with methods. An instance would therefore be passed either as a parameter on every getter, which is the per-call passing the ``ContextVar`` exists to remove and @@ -243,8 +243,7 @@ the prose being consolidated. The ``**queryables`` clause above originally named ``session`` among the rejected names. It was corrected after the fact: ``session`` holds no secret, so refusing it with a credentials message told callers the wrong thing, -and as a substring it claimed part of a namespace the *server* owns -- any -future query parameter containing it would have been unreachable behind that -message. ``dataretrieval/credentials.py`` records the exclusion at the +and as a substring it reserved part of a namespace the *server* defines -- any +future query parameter containing it would have been blocked by that message. ``dataretrieval/credentials.py`` records the exclusion at the predicate. The decision the clause makes -- credential-shaped names never reach a URL -- is unchanged. diff --git a/docs/source/architecture/decisions/0010-adapter-scoped-settings.rst b/docs/source/architecture/decisions/0010-adapter-scoped-settings.rst index 9b417191c..4dd65cd71 100644 --- a/docs/source/architecture/decisions/0010-adapter-scoped-settings.rst +++ b/docs/source/architecture/decisions/0010-adapter-scoped-settings.rst @@ -24,7 +24,7 @@ Context ADR 0009 resolved every setting through one flat namespace, on the premise that "a setting means the same thing to every service; services differ in the value -they want, not the vocabulary." Surveying the seven APIs this package retrieves +they use, not the vocabulary." Surveying the seven APIs this package retrieves from shows the premise is false. The settings themselves differ: .. list-table:: @@ -74,12 +74,12 @@ from shows the premise is false. The settings themselves differ: ``concurrency`` and ``parallel_chunks`` are meaningless for the four single-shot adapters -- there is nothing to fan out. ``ssl_check`` applies to four adapters (``waterdata``, ``nwdc``, ``nwis``, ``wqp``) and is currently a -per-call keyword outside the chain entirely; it reaches ``httpx``'s ``verify``, +per-call keyword outside the chain entirely; it is passed to ``httpx``'s ``verify``, verified by spying on the client. A flat namespace accepts ``configure(streamstats={"parallel_chunks": 8})`` without error, which is the typo class ADR 0009 exists to catch. -The credential is a separate axis, and measurement settled it. Probing the live +The credential is a separate axis, and measurement shows this. Probing the live APIs with and without a key: * NGWMN and Water Data are served from the *same host* @@ -90,7 +90,7 @@ APIs with and without a key: (997, 996, 996, 994, 993, 992), so the two adapters share one quota pool. * Water Data's OpenAPI declares ``ApiKeyHeader``/``ApiKeyQuery``; NGWMN's declares no security scheme at all across 34 paths -- yet the gateway meters - it regardless. Every response carries ``via: ... api-umbrella``. + it regardless. Every response includes ``via: ... api-umbrella``. The key is therefore a credential of the **gateway fronting the host**, not of either adapter. It cannot meaningfully vary per adapter: two keys against one @@ -128,15 +128,15 @@ Settings are scoped to the **adapter**, not the service, and not the host. does not replace it. Every setting still has a package-wide form, and the shipped ``API_USGS_*`` variables are package-wide by construction. ``retries`` and ``stall_timeout`` are additionally adapter-scopable, because - a service that responds slowly or refuses often warrants its own budget - without changing anyone else's. ``progress`` is not: it describes the + a service that responds slowly or returns errors often warrants its own + budget without changing the others'. ``progress`` is not: it describes the caller's terminal, and there is one progress line per call, so scoping it per adapter could only produce a contradiction. 3. **Precedence stays source-major.** Resolution checks the block, then the environment, then the file, as ADR 0009 defines; *within* each source an adapter-scoped value outranks a top-level one. The environment therefore - still outranks the file, so a stale adapter table cannot quietly override a + still outranks the file, so a stale adapter table cannot override a variable exported for one run. 4. **Adapter-scoped settings get no environment variables.** Every entry in @@ -164,7 +164,7 @@ Settings are scoped to the **adapter**, not the service, and not the host. 7. **Adapters are keyed by their service's name**, matching the module: ``waterdata``, ``ngwmn``, ``nwdc``, ``wqp``, ``nldi``, ``streamstats``. The deprecated ``nwis`` is deliberately absent: its calls pin - ``max_retries=0``, so a ``[nwis]`` table could only be reported as live and + ``max_retries=0``, so a ``[nwis]`` table could only be reported as in effect and then ignored -- the failure this decision exists to prevent. 8. **Each adapter is a named, typed parameter on** ``configure()``, annotated @@ -199,7 +199,7 @@ Consequences than in the leaf. - **A configuration object is still refused, but on narrower grounds than ADR - 0009 stated.** That ADR rejected an object because it had no way to *reach* + 0009 stated.** That ADR rejected an object because it had no way to be passed to the call. A per-adapter payload type does not have that problem -- the ``ContextVar`` remains the delivery mechanism and the type is only the payload's shape. ``TypedDict`` is chosen over a dataclass for the reason @@ -212,9 +212,9 @@ Consequences and the roster stops being duplicated. - **``show_configuration()`` gains a second section, not a matrix.** It prints - the top-level tier as today, then only those adapter overrides actually set. - A seven-by-eight grid of mostly-inherited values would obscure the answer to - "what will this call use". + the top-level tier as today, then only those adapter overrides set. + A seven-by-eight grid of mostly-inherited values would obscure which + value a call will use. - **The shared quota pool is not modelled.** ``[waterdata]`` and ``[ngwmn]`` appear to be independent settings but share one 1000/hour allowance. A host or @@ -232,8 +232,8 @@ Consequences - **``ssl_check`` stays a per-call argument and does not become a setting.** It is a defaulted keyword on 23 shipped getters across four adapters -- - ``wqp`` (9), ``nwis`` (10), ``waterdata`` (3) and ``nwdc`` (1) -- and it does - reach ``httpx``'s ``verify``. It was added in 2023 to what were then the only + ``wqp`` (9), ``nwis`` (10), ``waterdata`` (3) and ``nwdc`` (1) -- and it is + passed to ``httpx``'s ``verify``. It was added in 2023 to what were then the only modules; the OGC getters were added later and never adopted it, so its distribution records the package's history rather than a boundary. @@ -241,17 +241,17 @@ Consequences a per-call keyword it is a visible, scoped decision, while a config-file key or environment variable would make a security downgrade process-wide and invisible at the call site -- the opposite of the direction this chain - narrows everything else. It does not respect adapter boundaries: within + narrows everything else. It does not follow adapter boundaries: within ``waterdata`` it applies only to the getters that bypass the OGC engine, so - ``[waterdata] ssl_check`` would be applied by three getters and silently - ignored by the rest, exactly the pattern this ADR refuses elsewhere. And the + ``[waterdata] ssl_check`` would be applied by three getters and + ignored by the rest, the pattern this ADR refuses elsewhere. And the need it serves is already met better: the legitimate case is a TLS-intercepting corporate proxy, and ``httpx`` natively reads ``SSL_CERT_FILE`` and ``SSL_CERT_DIR`` on both its sync and async clients -- so that mechanism already covers *every* getter, including the OGC ones that have no ``ssl_check``, and it trusts the corporate CA rather than trusting - nothing. The ``bool`` type cannot even hold a CA bundle path, which is the - value a caller actually needs. + nothing. The ``bool`` type cannot hold a CA bundle path, which is the + value a caller needs. The configuration guide documents ``SSL_CERT_FILE`` for that case. Whether ``ssl_check`` should be deprecated outright is a public-API question left to diff --git a/docs/source/architecture/decisions/0011-configuration-profiles.rst b/docs/source/architecture/decisions/0011-configuration-profiles.rst index f5cae383f..05ec24739 100644 --- a/docs/source/architecture/decisions/0011-configuration-profiles.rst +++ b/docs/source/architecture/decisions/0011-configuration-profiles.rst @@ -20,8 +20,8 @@ Context ------- ADR 0010 gave each adapter its own table in the chain, so ``[ngwmn]`` narrows a -setting to NGWMN. That covers "tune one service" but not the case a -multi-service caller actually has: +setting to NGWMN. That covers tuning one service but not the case a +multi-service caller has: - **Several named configurations per adapter.** A caller with an overnight bulk configuration profile and a lower-rate daytime one for Water Data cannot @@ -36,7 +36,7 @@ multi-service caller actually has: Two further problems ADR 0010 left open bear on the same decision. The adapter roster is listed in four places, only one of which is derived -- adding an adapter needs coordinated edits, and forgetting one leaves a schema -no call site can reach, which happened to three adapters and shipped +no call site can use, which happened to three adapters and shipped undetected until a fitness test was written. And a setting's definition is in ``config`` rather than in the module that reads it, so adding a Water Data setting edits a file unrelated to Water Data. @@ -79,13 +79,13 @@ precedence rules do not order. Keyword settings are removed, so ``configure(api_key=...)`` no longer works. This is the most-typed line the feature exists to enable, and making it wordier -is a real cost, accepted deliberately so that every setting is passed the same +is a cost, accepted deliberately so that every setting is passed the same way. **Schemas are defined in their adapter; names are defined centrally.** ``configuration`` is a standard-library-only leaf every adapter may import, so it cannot import adapters. It holds the tuple of adapter *names*, which is what -parsing a file needs (is ``[ngwmn]`` a table or a typo?). Each adapter package +parsing a file needs (whether ``[ngwmn]`` is a table or a typo). Each adapter package owns its subclass, which is what a setting's definition needs to be local to the service that reads it. @@ -108,7 +108,7 @@ imported. Each level overrides the one below **per key**, so a named profile still inherits its adapter's default profile and the package-wide keys. Positions 1 and 2 are both code and both target one adapter, so the same-adapter rule -means they cannot tie. +means they cannot conflict. Position 2 above 3 inverts ADR 0009's environment-above-file rule for this one case. A profile named in code is a more deliberate act than a variable @@ -124,7 +124,7 @@ be defined in a module the parser cannot import. **Base URLs may be configured, from code only.** An adapter's configuration may include its base URL, settable in a ``configure()`` block and rejected from -the file and the environment. A file that silently redirects a data-retrieval +the file and the environment. A file that redirects a data-retrieval library to another host is a supply-chain hazard; an in-code block keeps the redirect where a reader sees it. @@ -134,7 +134,7 @@ and ADR 0009's rule reserving ``config`` as an abbreviation for the module and the file is withdrawn. The path has never been released, so no alias is needed. -**Credentials are unchanged, and measurement settled why.** The API key stays +**Credentials are unchanged, and measurement shows why.** The API key stays one package-wide setting scoped to the single host that accepts it. Probing the live services: @@ -168,11 +168,11 @@ anonymously today. The three hosts also keep independent counters, so ADR fields.** Which settings an adapter reads is the adapter's own concern, but what each setting *means* is shared, so the fields come from frozen mixin groups declared once beside their grammar. An adapter's configuration class -names the groups it composes and adds only what is genuinely its own. Declaring +names the groups it composes and adds only what is its own. Declaring ``retries: int | None = _UNSET`` directly in an adapter module satisfies this record literally while losing what it protects: the annotation would enforce nothing, could drift from the shared parser, and ``mypy --strict`` would not -notice, because it checks the annotation, not whether the field still matches +detect it, because it checks the annotation, not whether the field still matches the shared group. Consequences @@ -190,16 +190,16 @@ Consequences the same commit. - **``show_configuration()`` can only resolve the settings an adapter accepts once that adapter has been imported.** It names the adapters it could not - check rather than omitting them silently, which is the cost of lazy + check rather than omitting them, which is the cost of lazy validation. The *profile list* is not import-limited: what a profile is called is a fact about the file, so every ``[.]`` table it defines is listed, imported or not -- withholding one would make the - section's answer depend on which optional extras happened to be installed. + section's contents depend on which optional extras happened to be installed. - **Two names differ only by case** -- the ``configuration`` module and the ``Configuration`` class. The module stays out of the package's public exports, so ``from dataretrieval import configuration, Configuration`` cannot arise. - **Separate quota pools are still not modelled.** Three exist. Nothing in the - library needs to know yet. + library depends on them yet. - **``ssl_check`` is unaffected** and remains a per-call argument, for the reasons in ADR 0010. @@ -234,7 +234,7 @@ Satisfied. In ``tests/configuration_test.py``: - ``test_adapter_roster_names_real_modules_that_register_themselves`` and ``test_every_adapter_is_actually_wired_to_a_read_site`` -- the roster resolves, and no configuration exists that nothing reads. An adapter name - the code does not recognize now raises out of ``_resolve`` rather than + not in the roster now raises out of ``_resolve`` rather than falling through to the package-wide value, so the grep is a secondary check rather than the only one. @@ -250,10 +250,10 @@ Notes ``api.water.usgs.gov``. - Open, not decided here: whether ``parallel_chunks`` is renamed. ``fan_out`` was suggested and conflicts with the glossary, where fan-out is *executing* - chunks concurrently -- which ``concurrency`` already governs -- while + chunks concurrently -- which ``concurrency`` already controls -- while ``parallel_chunks`` instructs the planner to *divide* more finely. ADR 0009 rejected ``parallelism`` and ``chunk_parallelism`` for the same conflation. - ``chunk_count`` or ``target_chunks`` would stay on the correct side of it. + ``chunk_count`` or ``target_chunks`` would not conflate the two. - The setting-group clause was added after the original decision, consolidating under ADR 0000 a rule the configuration core was stating in prose. It does not change behavior. diff --git a/docs/source/architecture/decisions/0012-deprecation-horizons.rst b/docs/source/architecture/decisions/0012-deprecation-horizons.rst index 1bc89c237..a890de8a2 100644 --- a/docs/source/architecture/decisions/0012-deprecation-horizons.rst +++ b/docs/source/architecture/decisions/0012-deprecation-horizons.rst @@ -32,7 +32,7 @@ recorded in ``REMOVALS``. A deprecation advisory names three things: what is being removed, what to use instead, and the date on or after which it may be removed. The mechanism -tolerates an advisory with no date -- it then states no date rather +accepts an advisory with no date -- it then states no date rather than implying a schedule it does not have. A deprecation of a public name is expected to include one, and an advisory naming a replacement the caller cannot yet use is not finished. @@ -58,14 +58,14 @@ Consequences - A caller can see, from the warning alone, how long they have and what to migrate to. - Horizons can be audited and extended centrally, so a removal date cannot - arrive unnoticed in a module nobody is reading. + pass unnoticed in a module nobody is reading. - Deprecating something costs more than adding a ``warnings.warn`` call: the replacement must exist and a date must be chosen. That is the intended cost. - The package accumulates long-lived compatibility shims. This is accepted -- it is the cost of the compatibility characteristic, and the table makes the accumulation visible rather than hidden. - Nothing is removed on the horizon alone. A removal still needs a release that - says so. + states it. Compliance ---------- diff --git a/docs/source/architecture/decisions/0013-core-and-domain-terms.rst b/docs/source/architecture/decisions/0013-core-and-domain-terms.rst index 6177775d5..105b35d23 100644 --- a/docs/source/architecture/decisions/0013-core-and-domain-terms.rst +++ b/docs/source/architecture/decisions/0013-core-and-domain-terms.rst @@ -20,17 +20,17 @@ repeatedly: whether a docstring may say *site*, whether ``service=`` may name a collection, whether prose about NWIS is bound by a word chosen from the Water Data API. -The two sets behave differently because their authority differs. +The two sets are treated differently because their authority differs. Terms like *chunk*, *page*, *fan-out*, *plan*, *interruption*, *dialect* and *leaf* appear nowhere in any USGS API's vocabulary. They were invented here to describe mechanisms this package owns. Nothing external constrains them, so when the package spells one of them two ways -- the resolution chain's code said *tier* for what its founding records, ADRs 0009 and 0010, call a *source* -- -that is simply an inconsistency, and one that can be removed by deciding. +that is an inconsistency, and one that can be removed by deciding. Terms like *monitoring location* and *collection* are different. The services -name those things, and they do not agree with each other: +name those things, and they differ: .. list-table:: :header-rows: 1 @@ -51,7 +51,7 @@ name those things, and they do not agree with each other: - ``collection`` - ``collection`` -No decision here makes those agree. A caller who has read the WQP +No decision here makes those match. A caller who has read the WQP documentation looks for ``Station``; one reading Water Data's looks for ``monitoring_location_id``. An adapter that renamed either would be harder to use, not easier, and the parameter names are public surface besides. @@ -66,8 +66,8 @@ Decision The glossary holds two kinds of term, and they impose different obligations. -**Core terms are ours.** The package invented them and no service has a claim -on them: everything under *Retrieval*, *Failure and resumption*, *Configuration* +**Core terms are ours.** The package invented them and no service defines +them: everything under *Retrieval*, *Failure and resumption*, *Configuration* and *Boundaries*, plus *Collection family* and *Metadata*. One spelling, enforced everywhere it appears -- prose, identifiers, tests. A second spelling of a core term is a defect, not a variation, and is fixed rather than recorded. @@ -81,9 +81,9 @@ chooses one term for **prose**, so that documents about the package are internally consistent. It does not choose the names used in requests, and it does not choose for an adapter's public surface: each adapter keeps its own service's spelling in its parameters, and reproduces that service's vocabulary -faithfully where it appears in returned data. +where it appears in returned data. -An adapter is where the two meet. Its public surface uses its service's +An adapter uses both kinds. Its public surface uses its service's terms; what it passes to the shared modules uses the core terms. The translation is the adapter's responsibility, and a divergence at that boundary is the design working as intended rather than a defect. @@ -114,9 +114,9 @@ Consequences its definition rather than a list of exceptions. - A glossary entry now has an obligation to say which kind it is. That is a small cost per term and the reason the distinction is usable at all. -- The package's own inconsistencies in core vocabulary become defects with a - deadline rather than curiosities. The resolution chain's ``tier``-for-*source* - identifiers are the current example. +- The package's own inconsistencies in core vocabulary become defects to fix + rather than variations to tolerate. The resolution chain's + ``tier``-for-*source* identifiers are the current example. Compliance ---------- diff --git a/docs/source/architecture/index.rst b/docs/source/architecture/index.rst index 28949674f..a71518f04 100644 --- a/docs/source/architecture/index.rst +++ b/docs/source/architecture/index.rst @@ -27,8 +27,8 @@ are necessary: #. **Public API compatibility.** Established imports, function signatures, return shapes, metadata, warnings, and exception types should remain stable; intentional changes follow the project's deprecation policy. -#. **Correctness and data integrity.** Pagination and fan-out must not silently - return truncated or duplicated data after a failure. +#. **Correctness and data integrity.** Pagination and fan-out must not + return truncated or duplicated data after a failure as if complete. #. **Resilience.** Retry, rate-limit, interruption, and resume behavior must be explicit and bounded. Capabilities may differ where upstream protocols do. #. **Maintainability.** Modules should have cohesive responsibilities and @@ -58,7 +58,7 @@ The package is the layer between Python callers and remote hydrologic services:: The remote services own their schemas, paging mechanisms, rate limits, and availability. The library adapts those differences to documented Python -contracts but does not hide meaningful service-specific behavior. +contracts but does not conceal meaningful service-specific behavior. Composition and dependency view ------------------------------- @@ -131,7 +131,7 @@ Shared components callback, owning bounded concurrency, deterministic failure precedence, sparse completion state, resume, and the progress line. It is also the one entry point from synchronous getter code into the async internals: a query - with nothing to divide runs as a one-item fan-out rather than crossing a + with nothing to divide runs as a one-item fan-out rather than through a separate bridge. Internally, ``liveness`` is a stdlib-only leaf recording when data last arrived, so the page loop that observes progress and the retry loop that acts on it depend on ``liveness`` rather than on each other. @@ -152,7 +152,7 @@ Shared components ``BaseMetadata``, the second half of every getter's ``(DataFrame, metadata)`` return contract. A dependency-free leaf: nearly every service module needs this class, and while it was in ``utils`` beside the legacy - query code, importing it pulled in that module's whole HTTP stack + query code, importing it imported that module's whole HTTP stack transitively. The implementation module is private; the established public class path remains ``dataretrieval.utils.BaseMetadata``. @@ -163,7 +163,7 @@ Shared components ``dataretrieval.utils`` Data-shaping helpers, plus compatibility imports for names that - historically lived here (including ``Ambient``, ``BaseMetadata``, ``query`` + were historically defined here (including ``Ambient``, ``BaseMetadata``, ``query`` and ``to_str``, so their original import paths keep working). OGC does not depend on this legacy module; by default, do not add new service-specific behavior there. @@ -171,10 +171,10 @@ Shared components ``dataretrieval._querying`` The one-shot HTTP query path the single-request adapters (``nwis``, ``wqp``, ``nldi``, ``streamstats``, ``nwdc``) use: compose the URL, send - it, map the status, retry a transient. It left ``utils`` because the two - halves shared only a filename -- this one depends on ``exceptions`` and - ``transport``, the shaping half on ``codes`` and pandas, and no caller - wanted both. The implementation module is private; the established public + it, map the status, retry a transient. It was moved out of ``utils`` because the two + halves were in one file for no other reason -- this one depends on + ``exceptions`` and ``transport``, the shaping half on ``codes`` and + pandas, and no caller used both. The implementation module is private; the established public function paths remain ``dataretrieval.utils.query`` and ``dataretrieval.utils.to_str``. @@ -190,7 +190,7 @@ The intended direction is:: Dependencies must not point from shared infrastructure back to a public service adapter. ``.importlinter`` declares this as a layer stack and ``lint-imports`` checks it over the transitive import graph, so a violation routed through an -intermediary fails as surely as a direct one. The stack is exhaustive: a new +intermediary fails like a direct one. The stack is exhaustive: a new top-level module fails the contract until it is placed, so where a module belongs is decided when it is added rather than inferred later. @@ -225,7 +225,7 @@ The library preserves meaningful upstream differences rather than forcing every service into one return shape: - Water Data, NGWMN, and Water Use tabular getters return ``(DataFrame, - BaseMetadata)``. Geometry-bearing Water Data and NGWMN results may use a + BaseMetadata)``. Water Data and NGWMN results that include geometry may use a ``GeoDataFrame`` in the first position when geopandas is installed. ``BaseMetadata`` holds request URL, elapsed query time, response headers, and comments where the upstream format provides them. @@ -295,8 +295,8 @@ architecturally is the behavior around them: defaults to four. Backoff is exponential with full jitter and waits for bounded ``Retry-After`` values. Only failures that may not recur on a later attempt are re-sent: 429 and gateway 5xx, not a 500 rejecting the query - itself, and not a transport failure that is settled before the request - leaves (unresolvable host, unsupported scheme). Deprecated NWIS + itself, and not a transport failure that is determined before the request + is sent (unresolvable host, unsupported scheme). Deprecated NWIS compatibility paths do not opt in. ``API_USGS_STALL_TIMEOUT`` @@ -306,10 +306,10 @@ architecturally is the behavior around them: time: without this bound, four retries of a request that times out after a minute add up to four minutes without data. Progress restarts the budget -- a page received, or a queued chunk acquiring its concurrency slot. Neither a - slow but productive download nor the last chunks of a large fan-out are cut - short, and an attempt already in flight is never interrupted. This bound - never withholds the first retry, so one slow attempt cannot disable retry by - itself; after that, the budget decides whether to continue. A dead + slow but productive download nor the last chunks of a large fan-out are + stopped early, and an attempt already in flight is never interrupted. This bound + never blocks the first retry, so one slow attempt cannot disable retry by + itself; after that, the budget determines whether retrying continues. A dead connection therefore costs about two read timeouts rather than five full attempts. diff --git a/docs/source/meta/contributing.rst b/docs/source/meta/contributing.rst index 5a862a3ec..0b634203b 100644 --- a/docs/source/meta/contributing.rst +++ b/docs/source/meta/contributing.rst @@ -3,7 +3,7 @@ Contributing Contributions to ``dataretrieval`` are welcome. The repository's contributor requirements and development commands are in `CONTRIBUTING.md`_. That file is -the single source of truth for issue reports, change proposals, pull requests, +the one authoritative source for issue reports, change proposals, pull requests, coding standards, testing, documentation, and releases. For the design constraints that apply to architecturally significant changes, diff --git a/docs/source/reference/exceptions.rst b/docs/source/reference/exceptions.rst index 7a104db89..f69cc4285 100644 --- a/docs/source/reference/exceptions.rst +++ b/docs/source/reference/exceptions.rst @@ -20,7 +20,7 @@ are importable from the top level, e.g. class object under the name it was first published as -- so ``except ChunkInterrupted`` and ``except FanOutInterrupted`` are the same handler. The base class is named for the fan-out rather than for chunking because a Water Use -call fans out without dividing anything: the NWDC simply accepts one location +call fans out without dividing anything: the NWDC accepts one location per request. .. autoclass:: dataretrieval.FanOutInterrupted diff --git a/docs/source/userguide/configuration.rst b/docs/source/userguide/configuration.rst index e462efaf9..d6a5d1028 100644 --- a/docs/source/userguide/configuration.rst +++ b/docs/source/userguide/configuration.rst @@ -22,8 +22,8 @@ authenticates to a gateway rather than to an adapter, so it stays package-wide. One block, several services --------------------------- -This is the case the mechanism exists for. Say the file holds what you would -write once and keep — the key, a retry budget, and Water Data's everyday +This is the case the mechanism exists for. Suppose the file holds what you would +write once and keep — the key, a retry budget, and Water Data's usual concurrency — plus two named profiles for the settings you only sometimes want: @@ -131,7 +131,7 @@ Settings - the service's own - *(none — code only)* - Where to send one service's requests. Per adapter, and settable only in - a ``configure`` block: a file that silently redirected the library to + a ``configure`` block: a file that redirected the library to another host would be a supply-chain hazard. See :ref:`configuration-redirect`. @@ -160,14 +160,14 @@ for one adapter raises, so they cannot disagree inside one block. Between nested blocks the innermost decides, as it does for everything else. Precedence applies **per setting**. An environment that sets only -``API_USGS_PAT`` leaves a file-provided ``concurrency`` fully in effect — +``API_USGS_PAT`` leaves a file-provided ``concurrency`` in effect — sources are merged, not replaced. A variable that is *set but empty* (``export API_USGS_PAT=``, or a CI secret that resolves to nothing) does not count as configured, so an empty variable -your tooling happened to create cannot silently discard the key in your config +your tooling happened to create cannot discard the key in your config file. The one exception is ``API_USGS_PROGRESS``, where blank has always meant -"off" and so is treated as a real value. +"off" and so is treated as a set value. .. note:: @@ -188,7 +188,7 @@ file. The one exception is ``API_USGS_PROGRESS``, where blank has always meant An environment variable ----------------------- -Still fully supported, and the simplest option for a single key on one +Still supported, and the simplest option for a single key on one machine: .. code-block:: bash @@ -226,7 +226,7 @@ Any setting can go in the file: concurrency = 16 retries = 8 -Point ``DATARETRIEVAL_CONFIG`` at a different path to override the location — +Set ``DATARETRIEVAL_CONFIG`` to a different path to override the location — useful for a container or a job scheduler that mounts secrets elsewhere. @@ -262,7 +262,7 @@ otherwise: an adapter-scoped value outranks a package-wide one only within the same source, so ``API_USGS_CONCURRENT`` exported for one run still outranks a ``[ngwmn] concurrency`` in the file. -Between ``configure`` blocks that tie-break applies per block: an adapter +Between ``configure`` blocks that ordering applies per block: an adapter configuration outranks a package-wide value set by the *same* block, while anything set by a block nested inside it overrides both. So a ``configure(Configuration(concurrency=1))`` can still throttle a call an @@ -272,7 +272,7 @@ Each adapter accepts only the settings it reads, and they are the fields of its configuration class — ``concurrency`` and ``parallel_chunks`` are meaningless to an adapter that issues a single request, so ``StreamstatsConfiguration`` has no such field and ``[streamstats] parallel_chunks = 8`` is an error rather than a -line that quietly does nothing: +line that does nothing: ==================================== ====================================== ======================================== Adapter Configuration Accepts @@ -328,8 +328,8 @@ to settings you did not ask for. What comes back is inert until you pass it to since selecting one is something your code did. A profile holds settings and nothing else: ``[waterdata.bulk-pull.ngwmn]`` is -not a Water Data profile containing NGWMN detail, and selecting it says so -rather than quietly ignoring the nested table. Two adapters means two profiles, +not a Water Data profile containing NGWMN detail, and selecting it raises +rather than ignoring the nested table. Two adapters means two profiles, selected in the same block, as in :ref:`the example above `. @@ -364,8 +364,8 @@ what makes it package-wide. earlier form and are no longer accepted; write ``Configuration(api_key=...)`` and ``NgwmnConfiguration(concurrency=4)`` instead. Passing anything that is not - a configuration raises and names the replacement, so an old script says what - to write rather than failing obscurely. + a configuration raises and names the replacement, so an old script fails with a message + that states what to write. Because it is backed by a :class:`~contextvars.ContextVar`, the value applies to the current thread and to asyncio tasks started inside the block, and @@ -426,7 +426,7 @@ the ``bulk`` profile selected for the block: parallel_chunks 1 built-in default stall_timeout 60s built-in default - A built-in default is package-wide. An adapter may prefer its own for + A built-in default is package-wide. An adapter may use its own default for its own calls; a value from any source above overrides both. adapter overrides @@ -440,27 +440,27 @@ the ``bulk`` profile selected for the block: not reported: nldi (not imported, so the settings each accepts are unknown here) Each line names the exact origin, including which table inside the file, which -is usually enough to answer "why is it still using my old key?". A value that +is usually enough to find why a call is still using an old key. A value that came from a profile names the profile — ``configure() block -[waterdata.bulk]``, not merely "a block" — so a report taken from inside a -``with`` block says which selection produced it. Only settings actually +[waterdata.bulk]``, not only "a block" — so a report taken from inside a +``with`` block says which selection produced it. Only settings overridden for an adapter get a row in the second section; everything else is inherited from the rows above it. The profile section lists what the *file* defines, whether or not this run selected any of it. A named profile does nothing until a caller selects it, so -seeing ``[waterdata.bulk]`` there while no row above mentions it is the answer -to "I added a profile and nothing changed". +seeing ``[waterdata.bulk]`` there while no row above mentions it explains +why adding a profile changed nothing. The last line is the cost of validating an adapter's settings lazily: ``dataretrieval`` cannot say what ``nldi`` accepts until something imports it, -so it says that rather than quietly omitting the service. It is named rather -than left out, because an omitted service would read as "nothing is configured -for it", which is a different claim. +so it reports that rather than omitting the service. It is named rather +than left out, because an omitted service would imply that nothing is configured +for it, which is a different claim. It never raises. A malformed file or a value that fails its grammar is reported in place — on the ``config file`` line for a whole-file problem, or in that -setting's own row — because a broken configuration is exactly when you need +setting's own row — because a broken configuration is when you need this. @@ -487,7 +487,7 @@ or as a baseline in the config file — deliberately written, and visible in ``show_configuration()``. Put it in a ``[.]`` table rather than at the top level: a named profile applies only to runs that select it, while a top-level value applies to every query in every process that reads the file, -which is how a setting added for one bulk pull quietly exhausts an hourly quota +which is how a setting added for one bulk pull exhausts an hourly quota months later. ``dataretrieval`` warns if it finds one at the top level. The value limits optional refinement only. URL-byte safety can require more @@ -497,13 +497,13 @@ stays a single request. ``parallel_chunks(n)`` is shorthand for ``configure(Configuration(parallel_chunks=n))``: one scoping mechanism, so the innermost block takes precedence whichever form set it, and -``show_configuration()`` always reports the value the chunker will actually use. +``show_configuration()`` always reports the value the chunker will use. .. _configuration-redirect: -Pointing an adapter at another host ------------------------------------ +Redirecting an adapter to another host +-------------------------------------- ``base_url`` sends one adapter's requests somewhere else — a staging instance, a mirror, or a recording proxy — for the duration of a block: @@ -532,12 +532,12 @@ each raise a ``ConfigurationError`` saying the setting *may only be set in code, in a configure() block* and naming the configuration to pass it on instead. -A file or a shell export that silently redirected a data-retrieval library to +A file or a shell export that redirected a data-retrieval library to another host would be a supply-chain hazard: nothing at the call site would show it, and a script that reads correctly would be sending requests to someone else's service. A ``with`` block keeps the redirect where a reader of the script sees it. The refusal raises an error rather than ignoring the value, -for the same reason — a variable that was quietly ignored would leave you +for the same reason — a variable that was ignored would leave you believing you had redirected something. **The API key is not sent to the new host.** It is scoped to the one host that @@ -575,7 +575,7 @@ Behind a TLS-intercepting proxy ------------------------------- On a corporate network that re-signs HTTPS traffic, requests fail with a -certificate-verification error. Point the standard OpenSSL variables at your +certificate-verification error. Set the standard OpenSSL variables to your organization's CA bundle: .. code-block:: bash @@ -590,7 +590,7 @@ package — including the OGC collection getters (``get_daily``, Prefer this to ``ssl_check=False``. That argument exists on some of the older getters and switches certificate verification *off* rather than trusting your -CA, so it accepts any certificate a network path offers — and it is not +CA, so it accepts any certificate a network path presents — and it is not available on the OGC getters at all. A CA bundle keeps verification on and works everywhere. diff --git a/docs/source/userguide/errors.rst b/docs/source/userguide/errors.rst index ba68ac8a5..73ca99f0d 100644 --- a/docs/source/userguide/errors.rst +++ b/docs/source/userguide/errors.rst @@ -35,7 +35,7 @@ read-anywhere fields, so you rarely need to import the specific subclasses: * ``.status_code`` -- the HTTP status, or ``None`` when the failure included no response (a connection error, an over-long URL, ...). -* ``.retry_after`` -- seconds the server asked you to wait (its ``Retry-After`` +* ``.retry_after`` -- seconds the server specified to wait (its ``Retry-After`` header), or ``None``. * ``.retryable`` -- ``True`` when re-issuing the same request might succeed (a 429 / 5xx, or a connection failure); ``False`` otherwise. @@ -114,7 +114,7 @@ chunk paginates, splitting a large result further costs little or no extra quota *as long as each chunk still spans many pages*. (Ten states pulled as one request then page nearly as many times as ten per-state requests would; a split that leaves each chunk only a page or two adds its partial -final page.) So if you *know* your pull is large, ask for a finer split with +final page.) If you know your pull is large, request a finer split with ``parallel_chunks(n)``: you get roughly the same pages in more, smaller chunks, which gives smoother progress, more even concurrency, and a smaller unit of retry/resume. ``parallel_chunks`` is a scoped ``with`` block, so diff --git a/docs/source/userguide/timeconventions.rst b/docs/source/userguide/timeconventions.rst index ae6029ff0..d8a5b5e42 100644 --- a/docs/source/userguide/timeconventions.rst +++ b/docs/source/userguide/timeconventions.rst @@ -50,7 +50,7 @@ them to any local timezone with the pandas ``.dt`` accessor. After conversion the timestamps have New York's offset — ``-05:00`` during standard time, or ``-04:00`` during daylight saving time, since New York is 4 or 5 hours behind UTC depending on the time of year. The first midnight-UTC -reading rolls back to the previous calendar day (``2024-02-29``) once shifted +reading falls on the previous calendar day (``2024-02-29``) once shifted into New York time. diff --git a/tests/architecture_test.py b/tests/architecture_test.py index 874065153..c098eb4d2 100644 --- a/tests/architecture_test.py +++ b/tests/architecture_test.py @@ -1,15 +1,15 @@ """Executable fitness functions that complement the dependency contracts. -Plain dependency direction -- who may import whom, and in which direction -- is -declared in ``.importlinter`` and checked by ``lint-imports`` in pre-commit and -CI. Those rules used to be asserted here too, and are not any more: one rule -enforced in two places is one rule that gets updated in one place. +Plain dependency direction -- which modules may import which, and in which direction -- +is declared in ``.importlinter`` and checked by ``lint-imports`` in pre-commit and CI. +Those rules used to be asserted here too, and are not any more: one rule enforced in two +places is one rule that gets updated in one place. What remains is everything a boundary checker cannot see, because an import -graph has no opinion about it: +graph does not record it: * which *symbols* cross a boundary, not just which modules (``ogc.engine``'s - compatibility surface, ``ogc.requests`` borrowing header policy but not the + compatibility surface, ``ogc.requests`` importing header policy but not the executing calls); * the declared public surface -- ``__all__`` presence, ownership, and the facade union; @@ -38,7 +38,7 @@ #: How many request-building names ``ogc.engine`` currently needs. A ceiling #: rather than an exact list allows renames and deletions without weakening the -#: rule that orchestration must not absorb request construction again. +#: rule that orchestration must not take over request construction again. _MAX_ENGINE_REQUEST_IMPORTS = 5 @@ -148,7 +148,7 @@ def _imports_from(path: Path, module: str) -> set[str]: """Names *path* imports from *module*, wherever the import appears. Walks the whole tree rather than only ``tree.body`` so a function-local - import can't slip past an import-surface rule. + import is not missed by an import-surface rule. """ tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) return { @@ -160,7 +160,7 @@ def _imports_from(path: Path, module: str) -> set[str]: def test_exceptions_has_no_runtime_third_party_dependency() -> None: - """The shared error-policy leaf must remain cheap and cycle-safe.""" + """The shared error-policy leaf must remain dependency-light and cycle-safe.""" imports = _runtime_imports(PACKAGE_ROOT / "exceptions.py") roots = {module.partition(".")[0] for module in imports} third_party = roots - sys.stdlib_module_names - {"dataretrieval"} @@ -180,15 +180,14 @@ def test_runtime_import_graph_is_acyclic() -> None: def test_config_is_a_standard_library_only_leaf() -> None: - """The two-module configuration subsystem must stay cheap and cycle-safe. - - ``dataretrieval.configuration`` remains the public runtime interface and may - depend on its private foundation, ``dataretrieval._configuration_core``. - The core may depend only on the package's dependency-free leaves, - ``dataretrieval._ambient`` and ``dataretrieval.exceptions``. Neither module - may reach adapters or runtime third-party packages. ``tomli`` remains the - one third-party exception: it is the ``tomllib`` backport used on Python - 3.10. + """The two-module configuration subsystem must stay dependency-light and cycle-safe. + + ``dataretrieval.configuration`` remains the public runtime interface and may depend + on its private foundation, ``dataretrieval._configuration_core``. The core may + depend only on the package's dependency-free leaves, ``dataretrieval._ambient`` and + ``dataretrieval.exceptions``. Neither module may import adapters or runtime + third-party packages. ``tomli`` remains the one third-party exception: it is the + ``tomllib`` backport used on Python 3.10. """ subsystem = { PACKAGE_ROOT / "configuration.py": { @@ -223,8 +222,8 @@ def test_config_is_a_standard_library_only_leaf() -> None: def test_engine_request_import_surface_does_not_grow() -> None: """Engine imports only request names it uses and may not grow a new hub. - An import that nothing uses is dead weight, and a used name past the cap - means request construction is migrating back into engine. + An import that nothing uses should be removed, and a used name past the cap + means request construction is moving back into engine. """ path = PACKAGE_ROOT / "ogc" / "engine.py" imported = _imports_from(path, "dataretrieval.ogc.requests") @@ -242,8 +241,8 @@ def test_engine_request_import_surface_does_not_grow() -> None: unused = sorted(imported - referenced) assert not unused, f"ogc.engine has unused request imports: {unused}" - # Every imported name must resolve in ``requests``; a stale import of a name - # that moved or was deleted fails as soon as engine loads. + # Every imported name must resolve in ``requests``; a stale import of a name that + # was renamed or deleted fails as soon as engine loads. from dataretrieval.ogc import requests as ogc_requests missing = sorted(name for name in imported if not hasattr(ogc_requests, name)) @@ -340,9 +339,9 @@ def test_transport_is_execution_policy_only() -> None: Terminal rendering (``progress``) and pandas result assembly (``combining``) are top-level leaves that transport reports *into* and returns *through*. - They lived here only because they had to leave ``ogc`` and this was the - nearest home; keeping them out is what makes "transport is HTTP execution - policy" a checkable claim rather than a description of a grab bag. + They were here only because they had to be moved out of ``ogc`` and this was the + nearest place; keeping them out is what makes "transport is HTTP execution + policy" a checkable claim. """ misplaced = { "dataretrieval/transport/progress.py", @@ -367,16 +366,16 @@ def test_transport_is_execution_policy_only() -> None: def test_credential_policy_has_one_definition() -> None: """Only ``dataretrieval.credentials`` may name the API-key host. - Attaching the key and stripping it back off have to agree about which host - is authorized; the way they stop agreeing is a second copy of the host - string. ``transport.http`` re-exports the predicate, it does not restate it. + Attaching the key and stripping it back off have to agree about which host is + authorized; they stop agreeing when a second copy of the host string is added. + ``transport.http`` re-exports the predicate, it does not restate it. """ host = "api.waterdata.usgs.gov" # Walked as AST string *values*, not as source text. A line-substring match # produces both false negatives and false positives: it missed the - # ``"https://…"`` form three modules use to spell the same authority, and it - # flagged docstring prose that merely names the service. Docstrings are - # excluded here (they are documentation, not a second source of truth) + # ``"https://…"`` form three modules use to write the same host, and it + # flagged docstring prose that only names the service. Docstrings are + # excluded here (they are documentation, not a second definition) # while every other literal -- bare host or full base URL -- counts. offenders: list[str] = [] for path in sorted(PACKAGE_ROOT.rglob("*.py")): @@ -460,7 +459,7 @@ def test_active_service_exports_are_explicit() -> None: ``_literal_exports`` raises when ``__all__`` is missing, so the call is the first assertion. The second is what keeps these modules from becoming re-export hubs: a name in ``__all__`` that the module does not define came - from somewhere else, and now has two public homes that can drift apart. + from somewhere else, and now has two public import paths that can drift apart. """ for relative in _EXPLICIT_EXPORT_MODULES: path = PACKAGE_ROOT / relative @@ -485,12 +484,12 @@ def test_each_family_getter_has_exactly_one_home() -> None: def test_api_facade_exports_exactly_the_family_union() -> None: - """The facade re-exports every family getter and invents none of its own. + """The facade re-exports every family getter and adds none of its own. - Derived from the families' own ``__all__`` rather than a frozen copy: a - hardcoded union is 19 more strings to edit per new getter, and it would still - pass if a family gained an export the facade forgot to re-export -- the one - thing worth catching here. + Derived from the families' own ``__all__`` rather than a frozen copy: a hardcoded + union is 19 more strings to edit per new getter, and it would still pass if a family + gained an export the facade forgot to re-export -- the one thing this test exists to + catch. """ families = set().union( *(_literal_exports(PACKAGE_ROOT / f) for f in _WATERDATA_FAMILIES) @@ -502,10 +501,10 @@ def test_waterdata_api_is_a_logic_free_compatibility_facade() -> None: """The facade re-exports; it does not run anything. Statement *kinds* are checked, not just ``def``/``class``. Scanning for - definitions alone let a module-level ``for`` loop live here that rewrote + definitions alone let a module-level ``for`` loop exist here that rewrote every re-exported getter's ``__module__`` -- code owned by the family modules, mutated from a file certified "logic-free". A docstring, imports, - and plain assignments are the whole legitimate vocabulary of a facade. + and plain assignments are the only statement kinds a facade may contain. """ path = PACKAGE_ROOT / "waterdata" / "api.py" tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) @@ -522,15 +521,15 @@ def test_waterdata_api_is_a_logic_free_compatibility_facade() -> None: def test_ogc_request_construction_does_not_execute_http() -> None: - """Building a request may borrow header policy, never the executing calls. + """Building a request may import header policy, never the executing calls. Two assertions, because the first alone was once true while the rule was broken: ``requests.py`` imported ``ogc.schema`` purely to forward a name, and ``ogc.schema`` calls ``transport.http.get`` -- so constructing a request - dragged in the executing path with this test still green. + imported the executing path with this test still passing. The edge is named explicitly rather than checked over the transitive graph. - A closure from ``ogc.requests`` reaches the whole package, because + A transitive closure from ``ogc.requests`` covers the whole package, because ``transport.retry`` imports ``dataretrieval`` for the progress reporter and the package ``__init__`` imports every service; a rule stated that way would be either vacuous or a list of exceptions. @@ -549,8 +548,8 @@ def test_ogc_request_construction_does_not_execute_http() -> None: def test_empty_result_shaping_consults_the_schema_endpoint() -> None: """``_deal_with_empty`` names columns from the collection schema. - That is a real network call on an empty result, so the dependency is worth - pinning deliberately rather than leaving it to be removed as dead weight. + That is a real network call on an empty result, so the dependency should be pinned + deliberately rather than leaving it to be removed as unused. """ assert "dataretrieval.ogc.schema" in _runtime_imports( PACKAGE_ROOT / "ogc" / "shaping.py" @@ -558,13 +557,13 @@ def test_empty_result_shaping_consults_the_schema_endpoint() -> None: def test_nwdc_does_not_reimplement_fan_out_orchestration() -> None: - """Water Use must drive its locations through the shared fan-out executor. + """Water Use must run its locations through the shared fan-out executor. It previously ran its own ``asyncio.gather`` with a private semaphore and a hand-copied failure-precedence rule, kept in sync with ``FanOut`` by a - comment. Two copies of that rule is how they drift, and the duplicate lost + comment. Two copies of that rule can drift apart, and the duplicate lost resume, progress, and the shared concurrency setting. Assert the duplication - cannot quietly return. + cannot return without failing this test. """ source = (PACKAGE_ROOT / "nwdc.py").read_text(encoding="utf-8") tree = ast.parse(source) @@ -585,31 +584,30 @@ def test_nwdc_does_not_reimplement_fan_out_orchestration() -> None: def test_ratings_drives_http_through_the_shared_executor() -> None: """Ratings must not issue one-off synchronous HTTP requests. - Its STAC page walk and per-feature downloads previously ran hand-rolled - sync loops over ``transport.http.get`` -- no retry, no stall budget, no - progress line, no resume, and N serial downloads. Both stages now drive - ``paginate``/``FanOut`` like every other multi-request path; assert the - direct sync entry points cannot quietly return. + Its STAC page walk and per-feature downloads previously ran hand-written sync loops + over ``transport.http.get`` -- no retry, no stall budget, no progress line, no + resume, and N serial downloads. Both stages now run through ``paginate``/``FanOut`` + like every other multi-request path; assert the direct sync entry points cannot + return without failing this test. """ transport_names = _imports_from( PACKAGE_ROOT / "waterdata" / "ratings.py", "dataretrieval.transport.http" ) offenders = transport_names - {"default_headers"} assert not offenders, ( - "ratings imports executing sync transport helpers instead of driving " + "ratings imports executing sync transport helpers instead of using " f"the shared executor: {sorted(offenders)}" ) def test_fan_out_plans_are_sized_and_repeatably_iterable() -> None: - """What ``FanOut`` needs of a plan, checked on both real plan types. - - ``FanOutPlan`` is the two standard protocols, so a ``list`` conforms with - no adapter class and ``isinstance`` against a ``runtime_checkable`` - protocol would prove only that the methods exist. What is actually - load-bearing and *not* guaranteed by the type is repeatability: resume - keys completed work by position, so a plan whose second pass differed -- - a generator mistaken for a collection, say -- would re-issue chunks that + """What ``FanOut`` requires of a plan, checked on both real plan types. + + ``FanOutPlan`` is the two standard protocols, so a ``list`` conforms with no adapter + class and ``isinstance`` against a ``runtime_checkable`` protocol would prove only + that the methods exist. What matters and is not guaranteed by the type is + repeatability: resume keys completed work by position, so a plan whose second pass + differed -- a generator mistaken for a collection, say -- would re-issue chunks that no longer match the completed positions. """ import httpx @@ -621,7 +619,7 @@ def _build(**args: object) -> httpx.Request: plans = [ ChunkPlan({"sites": ["a", "b"]}, _build, url_limit=8000), - # Water Use hands its request list straight to ``FanOut``. + # Water Use passes its request list directly to ``FanOut``. [httpx.Request("GET", "https://example.invalid/data")], ] for plan in plans: @@ -636,13 +634,13 @@ def _build(**args: object) -> httpx.Request: def test_adr_references_resolve_to_a_record() -> None: """Every ``ADR NNNN`` citation names a record that exists, in every venue. - ADR 0000 routes cross-cutting rationale into the decision records and asks - the code to cite rather than restate. A renumbered or deleted record has to - fail here rather than leave a dangling pointer for a reader to chase. + ADR 0000 places cross-cutting rationale in the decision records and requires the + code to cite rather than restate. A renumbered or deleted record has to fail here + rather than leave a dangling pointer. Scoped to every venue ADR 0000 names, not just the package: the glossary, the contributor guide, and the architecture docs cite records too, and a - pointer rots there too. + pointer goes stale there too. """ repo_root = PACKAGE_ROOT.parent decisions = repo_root / "docs" / "source" / "architecture" / "decisions" diff --git a/tests/configuration_test.py b/tests/configuration_test.py index 0b081ddc5..e4045b79c 100644 --- a/tests/configuration_test.py +++ b/tests/configuration_test.py @@ -29,17 +29,16 @@ WATERDATA_URL = "https://api.waterdata.usgs.gov/ogcapi/v0/collections/daily/items" -# Where the base-URL tests redirect to. A host the suite can never reach, so a +# Where the base-URL tests redirect to. A host the suite cannot connect to, so a # redirect that failed to apply shows up as an unmocked request rather than as # a real one. _MIRROR = "https://mirror.example/waterdata" _MIRROR_RE = re.compile(r"^https://mirror\.example/") _WATERDATA_RE = re.compile(r"^https://api\.waterdata\.usgs\.gov/") -# One committed page of the ``daily`` collection, shared with the Water Data -# suite. Real response shape rather than a hand-made stub, so a redirect is -# exercised through the same shaping the getters normally do; it carries no -# ``links``, so nothing paginates. +# One committed page of the ``daily`` collection, shared with the Water Data suite. Real +# response shape rather than a hand-made stub, so a redirect is exercised through the +# same shaping the getters normally do; it has no ``links``, so nothing paginates. _DAILY_PAGE = json.loads( (pathlib.Path(__file__).parent / "data" / "waterdata_ogc_fixtures.json").read_text() )["daily"] @@ -52,7 +51,7 @@ def config_file(tmp_path, monkeypatch): def write(text: str): path = tmp_path / "config.toml" path.write_text(text) - path.chmod(0o600) # keep the loose-permission warning out of the way + path.chmod(0o600) # suppress the loose-permission warning for env in configuration.ENV_VARS.values(): monkeypatch.delenv(env, raising=False) monkeypatch.setenv(configuration.CONFIG_PATH_ENV, str(path)) @@ -185,11 +184,11 @@ def test_block_accepts_ints_and_strings(): def test_configure_takes_configurations_and_nothing_else(): - """The argument is an object, so a stray mapping or keyword cannot pass. + """The argument is an object, so an unexpected mapping or keyword cannot pass. - ``configure(ngwmn={"concurrency": 2})`` was the earlier spelling, and it is - exactly what a reader of an old script will try. Naming the replacement in - the error is the difference between a two-minute fix and a search. + ``configure(ngwmn={"concurrency": 2})`` was the earlier spelling, and it is what a + reader of an old script will try. Naming the replacement in the error lets the + reader fix the call without searching the docs. """ with pytest.raises(configuration.ConfigurationError, match="configuration objects"): with dataretrieval.configure({"concurrency": 2}): @@ -206,7 +205,7 @@ def test_configure_takes_configurations_and_nothing_else(): def test_two_configurations_for_one_adapter_raise(): """They are the one pairing with no defined order between them. - Silently letting the last win would make a block's meaning depend on + Letting the last one take precedence would make a block's meaning depend on argument order, which nothing in the surrounding chain does. """ with pytest.raises(configuration.ConfigurationError, match="two configurations"): @@ -223,7 +222,7 @@ def test_two_configurations_for_one_adapter_raise(): ): pass - # Two *different* adapters in one block is the whole point of the feature. + # Two *different* adapters in one block is what the feature exists for. with dataretrieval.configure( WaterdataConfiguration(concurrency=2), NgwmnConfiguration(concurrency=8) ): @@ -249,12 +248,12 @@ def test_a_configuration_resolves_end_to_end(config_file, monkeypatch): def test_an_adapter_configuration_narrows_to_one_adapter(monkeypatch): - """The adapter is a property of the class, so nothing else moves.""" + """The adapter is a property of the class, so nothing else changes.""" monkeypatch.delenv("API_USGS_RETRIES") # pinned by the autouse fixture with dataretrieval.configure(NgwmnConfiguration(retries=1)): assert configuration.retries(adapter="ngwmn") == 1 - # Every other adapter, and the package-wide read, are untouched -- + # Every other adapter, and the package-wide read, are unchanged -- # including waterdata, which shares NGWMN's host and its API key. for other in ("waterdata", "nwdc", "wqp", "streamstats"): assert configuration.retries(adapter=other) == configuration.DEFAULT_RETRIES @@ -308,7 +307,7 @@ async def main() -> list[str | None]: def test_named_profile_is_selected_in_code(config_file): - """``[.]`` reaches the chain only when a caller loads it.""" + """``[.]`` enters the chain only when a caller loads it.""" config_file( 'api_key = "shared"\nconcurrency = 4\n\n' "[waterdata]\nretries = 2\n\n" @@ -322,7 +321,7 @@ def test_named_profile_is_selected_in_code(config_file): assert configuration.concurrency(adapter="waterdata") is None # the profile assert configuration.retries(adapter="waterdata") == 2 # default profile assert configuration.api_key() == "shared" # package-wide, from the file - # It narrows to one adapter, so a sibling on the same host is untouched. + # It narrows to one adapter, so a sibling on the same host is unchanged. assert configuration.concurrency(adapter="ngwmn") == 4 assert configuration.concurrency(adapter="waterdata") == 4 @@ -331,9 +330,8 @@ def test_named_profile_is_selected_in_code(config_file): def test_a_code_selected_profile_outranks_the_environment(config_file, monkeypatch): """ADR 0011 inverts ADR 0009's environment-above-file rule for this case. - A profile named in code is a more deliberate act than a variable inherited - from a shell, and losing to that variable is what a caller would file a bug - about. + A profile named in code is a more deliberate act than a variable inherited from a + shell, and being overridden by that variable is what a caller would report as a bug. """ config_file("[waterdata.gentle]\nconcurrency = 2\n") monkeypatch.setenv("API_USGS_CONCURRENT", "16") @@ -362,8 +360,7 @@ def test_a_named_profile_layers_per_key_over_the_rungs_below(config_file): """Selecting a profile replaces keys, never whole rungs. Every rung overrides the one below it *per key* (ADR 0011), so one - adapter-scoped read here draws each of its four settings from a - different table. + adapter-scoped read here takes each of its four settings from a different table. """ config_file( "concurrency = 16\nretries = 3\nstall_timeout = 30\n\n" @@ -374,10 +371,9 @@ def test_a_named_profile_layers_per_key_over_the_rungs_below(config_file): with dataretrieval.configure(WaterdataConfiguration.load("bulk")): # the profile, over a package-wide key it names... assert configuration.concurrency(adapter="waterdata") is None - # ...the default profile, over a package-wide key the profile is silent - # about... + # ...the default profile, over a package-wide key the profile does not name... assert configuration.retries(adapter="waterdata") == 2 - # ...the package-wide key, which neither table touched... + # ...the package-wide key, which neither table set... assert configuration.stall_timeout(adapter="waterdata") == 30 # ...and a setting only the profile names. assert configuration.parallel_chunks(adapter="waterdata") == 8 @@ -410,7 +406,7 @@ def _resolved_settings() -> dict[object, object]: def test_adding_a_named_profile_changes_nothing_until_it_is_selected(config_file): """Inertness is what makes a profile safe to add to a file others share. - A named profile that could shift a setting on its own would make every + A named profile that could change a setting on its own would make every addition to a shared ``config.toml`` a change to every script reading it, which is the failure the retired global ``[profiles.]`` table had. """ @@ -424,7 +420,7 @@ def test_adding_a_named_profile_changes_nothing_until_it_is_selected(config_file ) assert _resolved_settings() == before - # ...and the profile does reach the chain once it is named in code, so the + # ...and the profile does enter the chain once it is named in code, so the # comparison above is inertness rather than a profile nothing can select. with dataretrieval.configure(WaterdataConfiguration.load("bulk")): assert configuration.parallel_chunks(adapter="waterdata") == 8 @@ -433,9 +429,9 @@ def test_adding_a_named_profile_changes_nothing_until_it_is_selected(config_file def test_a_named_profile_cannot_hold_a_nested_table(config_file): """``[waterdata.bulk.ngwmn]`` is the retired shape, not a deeper profile. - A profile carries settings for the one adapter it belongs to, so a table - inside one has no reading. Refused rather than skipped: silently dropping - it would leave the author believing they had tuned NGWMN. + A profile holds settings for the one adapter it belongs to, so a table inside one + has no meaning. Refused rather than skipped: dropping it without an error would + leave the author believing they had tuned NGWMN. """ config_file( "[waterdata.bulk]\nparallel_chunks = 8\n\n" @@ -456,7 +452,7 @@ def test_a_named_profile_cannot_hold_a_nested_table(config_file): def test_loading_an_undefined_profile_raises(config_file): - """A name the caller just typed is a typo, not a silent fall-through. + """A name the caller just typed is a typo, not a fall-through. The message lists what the file *does* define, because a misspelling is only recognizable next to the spelling that was meant -- and only for this @@ -476,8 +472,8 @@ def test_loading_an_undefined_profile_raises(config_file): assert "bulk, polite" in message assert "gentle" not in message - # An adapter with no profiles at all says so rather than trailing off after - # the colon, which would read as a truncated message. + # An adapter with no profiles at all says so rather than ending after the colon, + # which would read as a truncated message. config_file("[waterdata]\nconcurrency = 4\n") with pytest.raises(configuration.ConfigurationError, match="waterdata: none"): WaterdataConfiguration.load("bulk") @@ -568,7 +564,7 @@ def test_unknown_table_raises(config_file): def test_the_retired_profiles_table_names_its_replacement(config_file): """Nothing shipped with ``[profiles.]``, but the docs described it. - The generic "unknown table" message would send its author hunting for a + The generic "unknown table" message would leave its author looking for a typo in a table spelled exactly as they had been told to spell it. """ config_file("[profiles.bulk]\nconcurrency = 4\n") @@ -579,12 +575,12 @@ def test_the_retired_profiles_table_names_its_replacement(config_file): def test_the_retired_profile_environment_variable_is_ignored(config_file, monkeypatch): - """``DATARETRIEVAL_PROFILE`` went with the table it selected (ADR 0011). + """``DATARETRIEVAL_PROFILE`` was retired with the table it selected (ADR 0011). - A profile is now named in code. A variable exported once in a shell profile - and inherited by every subprocess is the opposite shape: invisible at the - call site, and able to switch every service at once. Honoring it under the - new grammar would restore exactly what the grammar removed. + A profile is now named in code. A variable exported once in a shell profile and + inherited by every subprocess is the opposite design: invisible at the call site, + and able to switch every service at once. Reading it under the new grammar would + restore what the grammar removed. """ config_file('concurrency = 4\n\n[waterdata.bulk]\nconcurrency = "unbounded"\n') monkeypatch.setenv("DATARETRIEVAL_PROFILE", "bulk") @@ -638,9 +634,9 @@ def test_explicit_config_path_is_expanded(monkeypatch): def test_relative_config_path_follows_the_working_directory(tmp_path, monkeypatch): """A relative ``DATARETRIEVAL_CONFIG`` is resolved against the *current* cwd. - The path memo keys on the working directory for exactly this reason: a + The path memo keys on the working directory for this reason: a scheduler or notebook that sets a relative path and chdirs per job would - otherwise keep serving the first job's credentials for the life of the + otherwise keep returning the first job's credentials for the lifetime of the process, with ``show_configuration()`` reporting the stale path as current. """ first = tmp_path / "first" @@ -800,10 +796,10 @@ def test_block_sourced_key_is_still_host_scoped(): def test_no_public_getter_accepts_a_credential_parameter(): """Guards the ``**queryables`` catch-all. - Every Water Data getter forwards unknown keywords as OGC query - parameters, so a getter that grew an ``api_key`` or ``session`` - parameter could serialize a credential into a URL. Credentials must - arrive through ``dataretrieval.configure`` instead. + Every Water Data getter forwards unknown keywords as OGC query parameters, so a + getter that grew an ``api_key`` or ``session`` parameter could serialize a + credential into a URL. Credentials must be supplied through + ``dataretrieval.configure`` instead. """ import inspect @@ -828,9 +824,9 @@ def test_no_public_getter_accepts_a_credential_parameter(): @pytest.mark.parametrize("allowed", ["session", "session_id", "sampling_session"]) def test_session_is_not_treated_as_a_credential(allowed): - """``session`` carries no secret, and the queryable namespace is the + """``session`` holds no secret, and the queryable namespace is the server's — a substring rule would make any future field containing it - unreachable behind a credentials message that misstates the problem.""" + blocked by a credentials message that misstates the problem.""" from dataretrieval.waterdata.utils import _flatten_queryables assert _flatten_queryables({"queryables": {allowed: 1}}) == {allowed: 1} @@ -849,7 +845,7 @@ def test_credential_keyword_cannot_enter_queryables(forbidden): ) -# --- wiring into the rest of the package --------------------------------- +# --- use by the rest of the package --------------------------------- def test_retry_policy_reads_the_block(): @@ -865,7 +861,7 @@ def test_parallel_chunks_baseline_comes_from_config(config_file): assert configuration.parallel_chunks() == 1 config_file("parallel_chunks = 8\n") assert configuration.parallel_chunks() == 8 - with parallel_chunks(2): # an explicit block still wins over the file + with parallel_chunks(2): # an explicit block still outranks the file assert configuration.parallel_chunks() == 2 assert configuration.parallel_chunks() == 8 @@ -873,8 +869,9 @@ def test_parallel_chunks_baseline_comes_from_config(config_file): def test_parallel_chunks_and_configure_share_one_mechanism(): """``parallel_chunks(n)`` is sugar for a package-wide ``Configuration``. - They must not be two competing scopes: whichever block is innermost wins, - so ``show_configuration()`` always reports the value the chunker will use. + They must not be two competing scopes: whichever block is innermost takes + precedence, so ``show_configuration()`` always reports the value the + chunker will use. """ from dataretrieval.ogc.chunking import parallel_chunks @@ -912,7 +909,7 @@ def test_blank_env_does_not_mask_the_config_file(config_file, monkeypatch): Container and CI tooling routinely materializes one (``docker run -e API_USGS_PAT`` with nothing to pass, a workflow secret absent on a fork). - Letting that outrank the file silently dropped the API key and sent every + Letting that outrank the file dropped the API key without an error and sent every request unauthenticated. """ config_file('api_key = "file-key"\nconcurrency = 4\nretries = 7\nprogress = true\n') @@ -938,7 +935,7 @@ def test_blank_progress_env_keeps_its_legacy_meaning(monkeypatch): def test_config_error_is_in_the_error_taxonomy(): - """A broken config surfaces from inside a getter, so it must be catchable.""" + """A broken config is raised from inside a getter, so it must be catchable.""" import dataretrieval.exceptions as exceptions assert issubclass(configuration.ConfigurationError, exceptions.DataRetrievalError) @@ -949,13 +946,13 @@ def test_config_error_is_in_the_error_taxonomy(): def test_show_config_reports_a_broken_file_instead_of_raising(config_file): - """The tool that explains a configuration must survive a broken one.""" + """The tool that explains a configuration must still run on a broken one.""" config_file("this is not = valid toml [[[\n") out = io.StringIO() dataretrieval.show_configuration(stream=out) # must not raise text = out.getvalue() assert "ERROR:" in text - # Every setting still gets a row rather than the report dying part-way. + # Every setting still gets a row rather than the report stopping part-way. for name in configuration.SETTINGS: assert name in text @@ -970,7 +967,7 @@ def test_show_config_reports_a_bad_value_in_its_own_row(monkeypatch): def test_top_level_parallel_chunks_warns(config_file): - """It spends quota in every process, so steer it into a profile.""" + """It spends quota in every process, so the warning recommends a profile.""" with pytest.warns(UserWarning, match="parallel_chunks"): config_file("parallel_chunks = 8\n") assert configuration.parallel_chunks() == 8 @@ -988,10 +985,10 @@ def test_non_regular_config_path_is_empty_configuration(monkeypatch): """``DATARETRIEVAL_CONFIG=/dev/null`` is how a run declares "no config". A non-regular path is treated as empty *without being opened*: settings are - re-resolved per request, so reading a stream would hand its contents to the - first getter and nothing to the rest (and a FIFO would block on open until - a writer appeared). Rejecting it would raise from ``_default_headers`` on - every request -- the opposite of what the caller asked for. + re-resolved per request, so reading a stream would give its contents to the first + getter and nothing to the rest (and a FIFO would block on open until a writer + appeared). Rejecting it would raise from ``_default_headers`` on every request -- + the opposite of what the caller asked for. """ monkeypatch.delenv("API_USGS_CONCURRENT", raising=False) monkeypatch.delenv("API_USGS_PAT", raising=False) @@ -1007,27 +1004,27 @@ def test_broken_config_does_not_break_unrelated_services(config_file): """A Water Data config problem must not fail a legacy NWIS/WQP call. Config resolution can raise, and ``_default_headers`` runs for every - service. Resolving the key only after the host check keeps the blast - radius on the calls that would actually receive it. + service. Resolving the key only after the host check keeps the failure + on the calls that would receive it. """ config_file("this is not = valid toml [[[\n") - # Legacy hosts never get the key, so they never touch the configuration. + # Legacy hosts never get the key, so they never read the configuration. assert "X-Api-Key" not in _default_headers("https://waterservices.usgs.gov/nwis/dv") assert "X-Api-Key" not in _default_headers("https://www.waterqualitydata.us/data") - # The authorized host still fails loudly rather than silently going out - # unauthenticated and hitting the anonymous rate limit. + # The authorized host still raises rather than sending the request + # unauthenticated under the anonymous rate limit. with pytest.raises(configuration.ConfigurationError): _default_headers(WATERDATA_URL) def test_default_config_path_follows_a_changed_home(tmp_path, monkeypatch): - """The default path derives from the home variable, so the memo watches it. + """The default path derives from the home variable, so the memo is keyed on it. Which variable that is depends on the platform: ``ntpath.expanduser`` reads ``USERPROFILE`` and ignores ``HOME``, so setting ``HOME`` on Windows - moves nothing and this asserted against the runner's real home directory. + changes nothing and this asserted against the runner's real home directory. """ home_var = "USERPROFILE" if os.name == "nt" else "HOME" monkeypatch.delenv(configuration.CONFIG_PATH_ENV, raising=False) @@ -1052,7 +1049,7 @@ def test_unselected_profile_is_not_validated(config_file): """An invalid value in a profile nobody selected must not fail every request. Profile tables are kept raw at parse time and validated only when one is - actually selected -- the same blast-radius rule ``_default_headers`` + selected -- the same isolation rule ``_default_headers`` follows for the key itself. """ config_file('api_key = "good"\n\n[waterdata.experimental]\nconcurrency = 0\n') @@ -1073,11 +1070,11 @@ def test_unknown_setting_in_an_unselected_profile_is_silent(config_file, recwarn def test_a_malformed_table_does_not_fail_another_adapters_call(config_file): - """The blast-radius rule, on the source a whole adapter table sits in. + """The isolation rule, on the source that holds a whole adapter table. Keys are checked when *that* adapter first resolves a setting, so an - invalid value in ``[nldi]`` costs a Water Data call nothing -- which is - also what lets an adapter's vocabulary live in a module this leaf cannot + invalid value in ``[nldi]`` does not affect a Water Data call -- which is + also what lets an adapter's vocabulary be defined in a module this leaf cannot import. """ config_file( @@ -1116,16 +1113,16 @@ def test_show_config_does_not_promise_a_built_in_default_holds_everywhere( ): """A row reading "built-in default" is package-wide, not a per-service claim. - ``concurrency`` resolves to 32 with nothing configured, but a Water Use call - uses that service's own preference of 4. The report is the tool for "what - will this actually use", so it must not let the reader take a package-wide - row as an answer for every service. + ``concurrency`` resolves to 32 with nothing configured, but a Water Use call uses + that service's own preference of 4. The report is the tool for finding what a call + will use, so it must not let the reader take a package-wide row as an answer for + every service. """ from dataretrieval import configuration from dataretrieval.nwdc import DEFAULT_CONCURRENT_REQUESTS # The suite pins API_USGS_CONCURRENT so dispatch is deterministic; clear it - # so the two kinds of default are what actually differ here. + # so the two kinds of default are what differ here. monkeypatch.delenv("API_USGS_CONCURRENT", raising=False) assert configuration.concurrency() != configuration.concurrency( DEFAULT_CONCURRENT_REQUESTS @@ -1134,7 +1131,7 @@ def test_show_config_does_not_promise_a_built_in_default_holds_everywhere( dataretrieval.show_configuration() out = capsys.readouterr().out assert "built-in default" in out - assert "An adapter may prefer its own" in out + assert "An adapter may use its own default" in out # --- adapter-scoped settings (ADR 0010) ---------------------------------- @@ -1144,18 +1141,20 @@ def test_adapter_table_overrides_the_top_level_per_setting(config_file): """A ``[ngwmn]`` table narrows one adapter, leaving the rest inherited.""" config_file("concurrency = 16\nretries = 3\n\n[ngwmn]\nconcurrency = 4\n") - # The adapter that asked for it gets it... + # The adapter the table names gets it... assert configuration.concurrency(adapter="ngwmn") == 4 # ...its sibling on the same host does not... assert configuration.concurrency(adapter="waterdata") == 16 - # ...and the package-wide read is untouched. + # ...and the package-wide read is unchanged. assert configuration.concurrency() == 16 # Per setting, not per table: retries still comes from the top level. assert configuration.retries(adapter="ngwmn") == 3 def test_one_block_configures_several_adapters(config_file): - """The requirement ADR 0009 deferred: gentle here, unchanged there.""" + """The requirement ADR 0009 deferred: a lower value for one adapter, unchanged + for the rest. + """ config_file("") with dataretrieval.configure( @@ -1169,7 +1168,7 @@ def test_one_block_configures_several_adapters(config_file): configuration.concurrency(adapter="waterdata") == configuration.DEFAULT_CONCURRENCY ) - # A package-wide value in the same block still reaches every adapter. + # A package-wide value in the same block still applies to every adapter. assert configuration.retries(adapter="ngwmn") == 7 @@ -1177,7 +1176,7 @@ def test_environment_outranks_an_adapter_table(config_file, monkeypatch): """Precedence is source-major: the env source is above the file source. Scope-major ordering would invert this the moment anyone added an adapter - table, so a variable exported for one run would lose to a stale file entry. + table, so a variable exported for one run would be overridden by a stale file entry. """ config_file("[ngwmn]\nconcurrency = 4\n") monkeypatch.setenv("API_USGS_CONCURRENT", "7") @@ -1201,7 +1200,7 @@ def test_adapter_rejects_a_setting_it_does_not_read(config_file): From code the refusal is a ``TypeError`` from the dataclass itself: the setting is not a field of ``WqpConfiguration``, so there is nowhere to put - it. That is the same refusal a type checker makes before the code runs. + it. That is the same rejection a type checker makes before the code runs. """ with pytest.raises(TypeError, match="concurrency"): WqpConfiguration(concurrency=2) @@ -1232,10 +1231,10 @@ def test_api_key_is_never_adapter_scoped(): def test_a_misspelled_setting_is_not_silently_swallowed(): """A typo must fail, not be accepted and ignored. - ``Configuration(concurrancy=8)`` is not a field, so the dataclass refuses - it by name -- taking it and dropping it would leave a caller believing a - setting is in force that no call reads, from a module whose job is to be - trustworthy about what a call will use. + ``Configuration(concurrancy=8)`` is not a field, so the dataclass rejects it by name + -- taking it and dropping it would leave a caller believing a setting is in force + that no call reads, from a module that exists to report accurately what a call will + use. """ with pytest.raises(TypeError, match="concurrancy"): Configuration(concurrancy=8) @@ -1244,10 +1243,9 @@ def test_a_misspelled_setting_is_not_silently_swallowed(): def test_adapter_roster_names_real_modules_that_register_themselves(): """Every name in the roster resolves to an adapter that owns a schema. - Two halves of one declaration: the roster is what parsing a file needs - (is ``[ngwmn]`` a table or a typo?), and the class is what validating that - table's keys needs. A name in one and not the other is a configuration - nothing could reach. + Two halves of one declaration: the roster is what parsing a file needs (is + ``[ngwmn]`` a table or a typo?), and the class is what validating that table's keys + needs. A name in one and not the other is a configuration nothing could use. """ import importlib @@ -1259,7 +1257,7 @@ def test_adapter_roster_names_real_modules_that_register_themselves(): def test_registering_an_adapter_outside_the_roster_raises(): - """The roster is the authority, so a class cannot invent an adapter.""" + """The roster is the authority, so a class cannot add an adapter.""" @dataclass(frozen=True) class BogusConfiguration(configuration.BaseConfiguration): @@ -1282,13 +1280,13 @@ def test_settings_for_an_unimported_adapter_is_not_an_error(monkeypatch): def test_every_adapter_is_actually_wired_to_a_read_site(): - """A schema nothing passes costs the caller a report they cannot trust. + """A schema nothing passes gives the caller a report that may be wrong. - ``show_configuration()`` would report a ``[nwis]`` override as live while - every call ignored it -- the report whose whole job is answering "what will - this call use" being confidently incorrect. Importability is the weaker - half of the invariant: it passed while ``waterdata.get_cql``, eight of nine - WQP getters, and all of ``nwis`` silently resolved package-wide. + ``show_configuration()`` would report a ``[nwis]`` override as live while every call + ignored it -- the report that exists to say what a call will use being wrong. + Importability is the weaker half of the invariant: it passed while + ``waterdata.get_cql``, eight of nine WQP getters, and all of ``nwis`` resolved + package-wide without an error. """ import pathlib @@ -1308,30 +1306,30 @@ def test_every_adapter_is_actually_wired_to_a_read_site(): def test_a_misspelled_adapter_at_a_read_site_raises(): """The other half of the invariant above, which a grep cannot check. - ``adapter="waterdatas"`` used to resolve *silently* package-wide: no table - matches the typo, every setting is accepted because nothing knows the - schema, and a ``[waterdata]`` table or a ``WaterdataConfiguration`` is then - ignored with nothing raised anywhere. The grep only sees that the correctly - spelled string occurs somewhere; it cannot see a second, misspelled one. + ``adapter="waterdatas"`` used to resolve package-wide without an error: no table + matches the typo, every setting is accepted because no schema is registered, and a + ``[waterdata]`` table or a ``WaterdataConfiguration`` is then ignored with nothing + raised anywhere. The grep only sees that the correctly spelled string occurs + somewhere; it cannot see a second, misspelled one. """ with pytest.raises(configuration.ConfigurationError, match="not a configurable"): configuration.retries(adapter="waterdatas") - # Every read site funnels through one resolver, so the check reaches them - # all -- including the accessors that would otherwise return a default. + # Every read site goes through one resolver, so the check covers them all -- + # including the accessors that would otherwise return a default. with pytest.raises(configuration.ConfigurationError, match="not a configurable"): configuration.base_url(adapter="nwis", default="https://example.invalid") def test_a_non_finite_stall_timeout_is_refused(): - """``inf`` parses as a float and silently disables the bound it sets.""" + """``inf`` parses as a float and disables the bound it sets.""" for bad in (float("inf"), float("nan")): with pytest.raises(configuration.ConfigurationError, match="finite"): Configuration(stall_timeout=bad) def test_stall_timeout_resolves_through_the_chain(config_file, monkeypatch): - """It was read straight from os.environ, so a block and the file were mute.""" + """It was read straight from os.environ, so a block and the file had no effect.""" config_file("stall_timeout = 15\n\n[wqp]\nstall_timeout = 300\n") assert configuration.stall_timeout() == 15 @@ -1347,9 +1345,9 @@ def test_stall_timeout_resolves_through_the_chain(config_file, monkeypatch): def test_base_url_applies_from_code_and_is_refused_from_the_file(config_file): """A redirect belongs where a reader of the script sees it (ADR 0011). - A configuration file that silently sent a data-retrieval library to another - host would be a supply-chain-shaped hazard, so the file refuses the setting - outright rather than accepting it and being trusted. + A configuration file that sent a data-retrieval library to another + host would be a supply-chain hazard, so the file refuses the setting + outright rather than accepting it. """ config_file("") @@ -1359,7 +1357,7 @@ def test_base_url_applies_from_code_and_is_refused_from_the_file(config_file): assert configuration.base_url(adapter="waterdata") == ( "https://mirror.example/ogcapi" ) - # It names one service, so it never reaches another. + # It names one service, so it never applies to another. assert configuration.base_url(adapter="ngwmn") is None assert configuration.base_url(adapter="waterdata") is None @@ -1375,7 +1373,7 @@ def test_base_url_applies_from_code_and_is_refused_from_the_file(config_file): def test_base_url_must_be_an_absolute_http_url(): - """A bare host would fail far from here, inside the request builder.""" + """A bare host would fail later, inside the request builder.""" with pytest.raises(configuration.ConfigurationError, match="absolute"): WaterdataConfiguration(base_url="mirror.example") with pytest.raises(configuration.ConfigurationError, match="absolute"): @@ -1383,12 +1381,12 @@ def test_base_url_must_be_an_absolute_http_url(): def test_base_url_is_refused_from_the_environment(monkeypatch): - """The environment is refused out loud, not merely unread. + """The environment is refused with an error, not only ignored. - ``API_USGS_BASE_URL`` is the spelling every other setting's variable - predicts, so a caller who exports it believes they have redirected - something. Leaving it out of ``ENV_VARS`` would make that belief false and - silent; the error names the block to write instead. + ``API_USGS_BASE_URL`` is the spelling every other setting's variable predicts, so a + caller who exports it believes they have redirected something. Leaving it out of + ``ENV_VARS`` would make that belief false with nothing to say so; the error names + the block to write instead. """ monkeypatch.setenv("API_USGS_BASE_URL", "https://evil.example") @@ -1396,7 +1394,7 @@ def test_base_url_is_refused_from_the_environment(monkeypatch): configuration.base_url(adapter="waterdata") # Refused even under a block that sets one, matching the file: the variable - # cannot work, and being quietly outranked is how it survives to a run where + # cannot work, and being outranked without notice is how it persists to a run where # nothing outranks it. Unsetting it is the only fix. with dataretrieval.configure(WaterdataConfiguration(base_url=_MIRROR)): with pytest.raises( @@ -1404,15 +1402,15 @@ def test_base_url_is_refused_from_the_environment(monkeypatch): ): configuration.base_url(adapter="waterdata") - # A configuration in this state is exactly what show_configuration() exists - # to explain, so it reports the failure rather than raising out of it. + # A configuration in this state is what show_configuration() exists to explain, so + # it reports the failure rather than raising out of it. out = io.StringIO() dataretrieval.show_configuration(stream=out) assert "only be set in code" in out.getvalue() def test_a_code_base_url_redirects_every_water_data_endpoint_family(httpx_mock): - """One Water Data configuration moves every endpoint family together.""" + """One Water Data configuration redirects every endpoint family together.""" httpx_mock.add_response(json=_DAILY_PAGE) httpx_mock.add_response(json={"data": []}) httpx_mock.add_response(json={"features": []}) @@ -1441,11 +1439,11 @@ def test_a_code_base_url_redirects_every_water_data_endpoint_family(httpx_mock): def test_a_code_base_url_redirects_the_adapters_requests(httpx_mock): - """The setting has to move real traffic, not just resolve to a string. + """The setting has to redirect real requests, not only resolve to a string. - Two adapters with unrelated request machinery -- the OGC engine and a plain + Two adapters with unrelated request paths -- the OGC engine and a plain one-shot GET -- because "the configuration reaches the request" is a claim - about each adapter's wiring, and one of them passing says nothing about the + about each adapter's request path, and one of them passing says nothing about the other. """ httpx_mock.add_response(method=None, url=_MIRROR_RE, json=_DAILY_PAGE) @@ -1456,7 +1454,7 @@ def test_a_code_base_url_redirects_the_adapters_requests(httpx_mock): redirected_url = str(httpx_mock.get_requests()[-1].url) # Nothing configured: back to the service's own base, so the redirect is - # scoped to the block rather than latched somewhere at import. + # scoped to the block rather than fixed at import. waterdata.get_daily(monitoring_location_id="USGS-05427718") direct_url = str(httpx_mock.get_requests()[-1].url) @@ -1472,12 +1470,12 @@ def test_a_code_base_url_redirects_the_adapters_requests(httpx_mock): def test_a_redirected_adapter_is_not_sent_the_api_key(httpx_mock): - """The key is scoped to the host that honors it, and a mirror is not it. + """The key is scoped to the host that accepts it, and a mirror is not it. - ``credentials.accepts_api_key`` is checked where the header is attached, so - a redirect needs no second rule to be safe -- but "needs no rule" is exactly - the kind of claim that stops being true silently, and the cost of it being - false is a credential handed to whatever host the block named. + ``credentials.accepts_api_key`` is checked where the header is attached, so a + redirect needs no second rule to be safe -- but "needs no rule" is the kind of claim + that can stop being true with no test detecting it, and the cost of it being false + is a credential sent to whatever host the block named. """ httpx_mock.add_response(method=None, url=_MIRROR_RE, json=_DAILY_PAGE) httpx_mock.add_response(method=None, url=_WATERDATA_RE, json=_DAILY_PAGE) @@ -1522,7 +1520,7 @@ def validate(self) -> None: def test_show_configuration_lists_only_real_overrides(config_file): - """A full adapter-by-setting grid would bury the answer in inherited rows.""" + """A full adapter-by-setting grid would obscure the value among inherited rows.""" config_file("concurrency = 16\n\n[ngwmn]\nconcurrency = 4\n") out = io.StringIO() @@ -1538,12 +1536,12 @@ def test_show_configuration_lists_only_real_overrides(config_file): def test_inner_block_can_lower_a_setting_an_outer_block_scoped(config_file): - """The innermost block wins across *both* scopes, not just within one. + """The innermost block takes precedence across *both* scopes, not only within one. An adapter-scoped value is the more specific of two written by the same block. It must not outrank one written by a block nested *inside* it, or - the documented recovery from QuotaExhausted -- wait, then re-issue more - gently -- cannot be expressed once any adapter table is in play. + the documented recovery from QuotaExhausted -- wait, then re-issue at + lower concurrency -- cannot be expressed once any adapter table is set. """ config_file("") @@ -1554,7 +1552,7 @@ def test_inner_block_can_lower_a_setting_an_outer_block_scoped(config_file): def test_adapter_scope_still_wins_within_one_block(config_file): - """Depth breaks ties between blocks, never within one.""" + """Depth decides between blocks, never within one.""" config_file("") with dataretrieval.configure( @@ -1591,22 +1589,21 @@ def test_parallel_chunks_block_survives_an_adapter_scoped_outer_block(): # 6 the adapter's built-in preference, passed by the adapter's own read site # 7 the package built-in default # -# The tests below walk it as a *chain*: each one knocks the rung above out and -# asserts the next takes over. Seven independent single-rung assertions would -# all still pass if two rungs collapsed into one, which is the mistake worth -# catching -- rungs 2 and 3 are the pair a refactor is most likely to fuse, -# since 2 above 3 is the one place ADR 0011 inverts ADR 0009. +# The tests below test it as a chain: each one removes the rung above and asserts the +# next takes over. Seven independent single-rung assertions would all still pass if two +# rungs collapsed into one, which is the mistake this exists to catch -- rungs 2 and 3 +# are the pair a refactor is most likely to merge, since 2 above 3 is the one place ADR +# 0011 inverts ADR 0009. # -# ``nwdc`` and ``concurrency`` are the pair that can express all seven. NWDC is -# the adapter that ships a built-in preference of its own -- 4 concurrent -# requests, because the service is only stress-tested that far -- distinct from -# the package default of 32, and that difference is the only way rungs 6 and 7 -# can be told apart at all. +# ``nwdc`` and ``concurrency`` are the pair that can express all seven. NWDC is the +# adapter that has a built-in preference of its own -- 4 concurrent requests, because +# the service is only stress-tested that far -- distinct from the package default of 32, +# and that difference is the only way rungs 6 and 7 can be distinguished. #: Rungs 5, 4 and 2, with a distinct value per rung so a resolved number #: identifies the table it came from. Top-level keys are written first because #: TOML assigns a bare key to whichever table header precedes it: moved below -#: ``[nwdc]``, ``concurrency = 15`` would quietly stop being a rung-5 key and +#: ``[nwdc]``, ``concurrency = 15`` would stop being a rung-5 key and #: become a second rung-4 one, and the tests would still pass by coincidence. _LADDER_FILE = ( "concurrency = 15\n" # rung 5: the package-wide keys @@ -1628,8 +1625,8 @@ def _nwdc_concurrency() -> int | None: """Resolve ``concurrency`` the way NWDC's own fan-out does. Through the adapter's read site rather than a bare ``concurrency()``, so - the built-in preference at rung 6 is really in the chain and the ladder is - exercised as the adapter experiences it. + the built-in preference at rung 6 is in the chain and the ladder is + exercised as the adapter resolves it. """ return configuration.concurrency(DEFAULT_CONCURRENT_REQUESTS, adapter="nwdc") @@ -1646,17 +1643,17 @@ def test_a_configuration_instance_tops_the_ladder(config_file, monkeypatch): def test_a_loaded_profile_beats_the_environment(config_file, monkeypatch): """Rung 2 over rung 3 -- the one inversion ADR 0011 exists to make. - ADR 0009 put the environment above the file, and a named profile lives in - the file, so those two rules alone predict that ``API_USGS_CONCURRENT`` in - the shell wins. It does not: what reaches the chain is the caller *naming* - the profile in code, which is a more deliberate act than a variable - inherited from whatever started the process, and losing to that variable - is the behaviour a caller would file a bug about. + ADR 0009 put the environment above the file, and a named profile is in the file, so + those two rules alone predict that ``API_USGS_CONCURRENT`` in the shell takes + precedence. It does not: what enters the chain is the caller *naming* the profile in + code, which is a more deliberate act than a variable inherited from whatever started + the process, and being overridden by that variable is the behaviour a caller would + report as a bug. The inversion is also bounded, which the second half asserts: it covers what the profile names and nothing else, so ``retries`` -- which the file sets at the top level and no selected profile mentions -- still follows the - original environment-above-file rule inside the very same block. + original environment-above-file rule inside the same block. """ config_file(_LADDER_FILE) monkeypatch.setenv("API_USGS_CONCURRENT", str(_LADDER_ENV)) @@ -1664,13 +1661,14 @@ def test_a_loaded_profile_beats_the_environment(config_file, monkeypatch): assert _nwdc_concurrency() == _LADDER_ENV # rung 3, until a profile is selected with dataretrieval.configure(NwdcConfiguration.load("tuned")): - assert _nwdc_concurrency() == 12 # rung 2 wins for the key it names... + assert _nwdc_concurrency() == 12 # rung 2 applies for the key it names... assert configuration.retries(adapter="nwdc") == 9 # ...and only that key - assert _nwdc_concurrency() == _LADDER_ENV # and the shell has it back on exit + # ...and the environment value applies again on exit. + assert _nwdc_concurrency() == _LADDER_ENV def test_the_environment_beats_the_adapters_default_profile(config_file, monkeypatch): - """Rung 3 over rung 4: the file's always-on table is still just the file.""" + """Rung 3 over rung 4: the file's always-on table is still only the file.""" config_file(_LADDER_FILE) monkeypatch.setenv("API_USGS_CONCURRENT", str(_LADDER_ENV)) @@ -1680,7 +1678,7 @@ def test_the_environment_beats_the_adapters_default_profile(config_file, monkeyp def test_the_adapters_default_profile_beats_the_package_wide_keys(config_file): - """Rung 4 over rung 5: within the file, the narrower table decides.""" + """Rung 4 over rung 5: within the file, the narrower table applies.""" config_file(_LADDER_FILE) assert _nwdc_concurrency() == 14 @@ -1689,11 +1687,11 @@ def test_the_adapters_default_profile_beats_the_package_wide_keys(config_file): def test_the_package_wide_keys_beat_the_adapters_built_in_preference(config_file): - """Rung 5 over rung 6: a user-written value outranks an adapter's taste. + """Rung 5 over rung 6: a user-written value outranks an adapter's preference. The adapter's preference is a default, not a cap. One able to override a - setting the user actually wrote would make that setting a lie -- so a - top-level key the user never scoped to NWDC still reaches NWDC's calls. + setting the user wrote would make that setting untrue -- so a + top-level key the user never scoped to NWDC still applies to NWDC's calls. """ config_file("concurrency = 15\n") @@ -1711,7 +1709,7 @@ def test_the_adapters_built_in_preference_beats_the_package_built_in_default( assert _nwdc_concurrency() == DEFAULT_CONCURRENT_REQUESTS assert DEFAULT_CONCURRENT_REQUESTS != configuration.DEFAULT_CONCURRENCY # It is the read site's own figure, not a property of the adapter, so a - # caller that states no preference lands on the package default instead -- + # caller that states no preference gets the package default instead -- # which is what makes rungs 6 and 7 two rungs rather than one. assert ( configuration.concurrency(adapter="nwdc") == configuration.DEFAULT_CONCURRENCY @@ -1721,7 +1719,7 @@ def test_the_adapters_built_in_preference_beats_the_package_built_in_default( def test_the_package_built_in_default_is_the_floor(config_file): """Rung 7: with the six rungs above it empty, every setting still resolves. - The floor is what makes the whole chain optional -- a caller who has + The lowest rung is what makes the whole chain optional -- a caller who has configured nothing at all gets working values rather than an error. """ config_file("") @@ -1742,9 +1740,9 @@ def test_the_package_built_in_default_is_the_floor(config_file): def test_the_top_two_rungs_cannot_tie(config_file): """Rungs 1 and 2 both target one adapter, so no block can hold both. - That is what stops the ladder needing a tie-break nobody could remember: - the same-adapter rule refuses the pairing where the order would matter, - and between *nested* blocks the ordinary rule applies -- the innermost + That is what sis the highest rung of the ladder needing a tie-break rule nobody + could remember: the same-adapter rule rejects the pairing where the order would + matter, and between *nested* blocks the ordinary rule applies -- the innermost decides, whichever kind of configuration it holds. """ config_file(_LADDER_FILE) @@ -1766,8 +1764,8 @@ def test_the_top_two_rungs_cannot_tie(config_file): # "Rung 1 above rung 2" is a claim about one adapter, so a *package-wide* # instance is not the thing it is talking about: it targets no adapter at # all. Alongside a loaded profile in one block the adapter-scoped value is - # the more specific of the two and wins for that adapter (ADR 0010), while - # the package-wide value still governs every other adapter. + # the more specific of the two and takes precedence for that adapter (ADR + # 0010), while the package-wide value still governs every other adapter. with dataretrieval.configure( Configuration(concurrency=_LADDER_INSTANCE), NwdcConfiguration.load("tuned") ): @@ -1778,9 +1776,9 @@ def test_the_top_two_rungs_cannot_tie(config_file): def test_load_returns_an_instance_carrying_only_the_profiles_keys(config_file): """``load`` is a constructor: it reads one table and returns the class. - Only what the table names is carried, so every other setting stays unset + Only what the table names is included, so every other setting stays unset and keeps inheriting from the rungs below rather than being pinned to a - default the profile never asked for. That is what makes a profile a + default the profile never named. That is what makes a profile a *contribution* to the chain rather than a replacement for it. """ config_file( @@ -1798,12 +1796,12 @@ def test_load_returns_an_instance_carrying_only_the_profiles_keys(config_file): # --- show_configuration() reports profiles -------------------------------- # -# The report exists to answer "why is this call using that value?", so every +# The report exists to explain where each value came from, so every # row names the source that supplied it. A value from a profile is the case a # bare "configure() block" label cannot distinguish: a configuration written in # code and -# one loaded from a table reach the chain by the same route, and only the -# latter has a name in a file the caller can go and read. +# one loaded from a table enter the chain the same way, and only the +# latter has a name in a file the caller can read. #: The file the documented sample is generated from. Exercises every section: #: package-wide keys, an adapter's default profile, and a named profile. @@ -1817,7 +1815,7 @@ def test_load_returns_an_instance_carrying_only_the_profiles_keys(config_file): ) #: The illustrative path the samples print, standing in for the temporary file -#: the test actually writes. Substituting it is the *only* edit made to the +#: the test writes. Substituting it is the *only* edit made to the #: captured output -- everything else has to match what the function printed. _SAMPLE_PATH = "/home/u/.dataretrieval/config.toml" @@ -1839,11 +1837,11 @@ def test_show_configuration_names_the_profile_a_value_came_from(config_file): """A value from a profile is reported with that profile, not with "a block". ``WaterdataConfiguration.load("bulk")`` and ``WaterdataConfiguration(...)`` - enter the chain by the same route and are indistinguishable once their + enter the chain the same way and are indistinguishable once their values are in the block, so a report that said only ``configure() block`` left a caller who selected a profile they did not intend -- or who had forgotten a profile was selected at all -- with nothing to look at. The - label is the table's own spelling, so it is greppable in the file that + label is the table's own spelling, so it can be searched for in the file that defines it. """ config_file("[waterdata.bulk]\nconcurrency = 6\n") @@ -1855,7 +1853,7 @@ def test_show_configuration_names_the_profile_a_value_came_from(config_file): assert "configure() block [waterdata.bulk]" in out.getvalue() # A configuration written in code has no profile to name, so it names its - # adapter alone rather than inventing one -- and the package-wide one + # adapter alone rather than adding one -- and the package-wide one # narrows to nothing, so it names neither. out = io.StringIO() with dataretrieval.configure( @@ -1875,10 +1873,10 @@ def test_show_configuration_names_the_profile_a_value_came_from(config_file): def test_a_loaded_profile_remembers_its_name_without_becoming_a_setting(config_file): """The profile name is provenance, so it is not a field and not a value. - Keeping it off the fields is what stops it reaching :meth:`settings`, the - ``configure()`` frame, and equality: two configurations carrying the same - settings stay interchangeable however each was spelled, which is what - makes a configuration a value rather than a record of how it was built. + Keeping it off the fields is what keeps it out of :meth:`settings`, the + ``configure()`` frame, and equality: two configurations holding the same settings + stay interchangeable however each was spelled, which is what makes a configuration a + value rather than a record of how it was built. """ config_file("[waterdata.bulk]\nconcurrency = 6\n") @@ -1896,15 +1894,15 @@ def test_show_configuration_lists_the_profiles_the_file_defines( ): """A named profile is inert until selected, so the file's are listed too. - "I added ``[waterdata.bulk]`` and nothing changed" is the confusion this - section exists for: the profiles are there, and no row above names one - because no caller selected one. A report that mentioned a profile only - once it had been selected would leave that silence unexplained. + A profile that was added and had no effect is the case this section exists for: the + profiles are there, and no row above names one because no caller selected one. A + report that mentioned a profile only once it had been selected would leave that lack + of effect unexplained. - Names are read from the parsed file, so an adapter this process never - imported still has its profiles listed: what a table *means* needs the - import, what it is called does not, and hiding it would make the section - depend on which optional extras happened to be installed. + Names are read from the parsed file, so an adapter this process never imported still + has its profiles listed: what a table *means* needs the import, what it is called + does not, and omitting it would make the section depend on which optional extras + happened to be installed. """ monkeypatch.delitem(configuration._REGISTRY, "nldi", raising=False) config_file( @@ -1921,7 +1919,7 @@ def test_show_configuration_lists_the_profiles_the_file_defines( assert listed == "[waterdata.bulk], [ngwmn.gentle], [nldi.gentle]" # The adapter's *default* profile is not a named one: it is always in - # effect and already shows up as a source, so listing it here is noise. + # effect and already shows up as a source, so listing it here adds nothing. assert "[ngwmn]" not in listed # Inert, and the report says so by never naming one as a source. assert "configure() block" not in text @@ -1931,7 +1929,7 @@ def test_show_configuration_reports_an_unimported_adapter(config_file, monkeypat """An adapter this process cannot report on is named, never omitted. NLDI is imported on demand for the geopandas extra, so a process that has - not touched it cannot say which settings it accepts -- the cost of + not imported it cannot say which settings it accepts -- the cost of validating an adapter's keys lazily (ADR 0011). Leaving it out of the report would read as "nothing is configured for nldi", which is a different claim from "this report could not check", and the caller cannot @@ -1950,8 +1948,8 @@ def test_show_configuration_reports_an_unimported_adapter(config_file, monkeypat # not be named as uncoverable. assert "waterdata" not in text.split("not reported:", 1)[1] - # The line is a statement about this process, not about nldi: once the - # module is imported its configuration registers and the caveat goes away. + # The line is a statement about this process, not about nldi: once the module is + # imported its configuration registers and the caveat is no longer printed. @dataclass(frozen=True) class _AsImported(configuration.BaseConfiguration): adapter: ClassVar[str] = "nldi" @@ -1965,16 +1963,17 @@ class _AsImported(configuration.BaseConfiguration): def test_show_configuration_sample_output_is_current(config_file, monkeypatch): - """The documented samples are this function's real output, not a drawing. + """The documented samples are this function's real output, not a hand-written + illustration. Both had drifted from it -- the docstring wrapped a line the function prints whole, the user guide had lost a paragraph -- because a sample kept - by hand is only ever as fresh as the last person who remembered it. So + by hand is only as current as its last manual update. So the scenario is rebuilt here and the output compared verbatim; the only edit is swapping the temporary path for the illustrative one. Regenerate by running this test and copying the reported ``actual`` into - both places, never by editing them to taste. + both places, never by editing them by hand. """ path = config_file(_SAMPLE_FILE) monkeypatch.setenv("API_USGS_RETRIES", "8") @@ -2006,13 +2005,14 @@ def test_show_configuration_sample_output_is_current(config_file, monkeypatch): def test_show_configuration_survives_a_malformed_profile(config_file): - """Explaining a broken configuration is the job, so nothing here validates. + """Explaining a broken configuration is what the report is for, so nothing + here validates. The section lists what the file *defines*; a profile's keys are checked when a - caller selects it. So a profile holding a value that fails its grammar -- - or the nested table a file migrated from the retired ``[profiles.]`` - layout still carries -- is reported rather than taking the report down - with it, which is the one moment a caller most needs it. + caller selects it. So a profile holding a value that fails its grammar -- or the + nested table a file migrated from the retired ``[profiles.]`` layout still + carries -- is reported rather than failing the report, which is when a caller most + needs it. """ config_file( '[waterdata.bulk]\nconcurrency = "nope"\n\n' @@ -2047,7 +2047,7 @@ def test_a_non_numeric_stall_timeout_is_rejected_by_type(self): def test_a_bool_is_not_a_number_of_seconds(self): """``True`` is an ``int`` in Python, so a bare isinstance check would - accept ``stall_timeout = true`` and silently mean one second.""" + accept ``stall_timeout = true`` and be read as one second with no error.""" with pytest.raises(configuration.ConfigurationError): _core._coerce_seconds(True, "stall_timeout", "") @@ -2083,8 +2083,9 @@ def test_an_adapter_key_that_is_not_a_table_says_what_it_should_be( class TestConfigPathResolutionFailures: - """The file layer sits on the per-request path, so a filesystem that will - not answer must not take every query down with it.""" + """The file layer is on the per-request path, so a filesystem that cannot be read + must not fail every query. + """ def test_an_unresolvable_home_leaves_the_file_layer_inert(self, monkeypatch): """A container with no passwd entry raises from ``Path.home()``. The @@ -2102,8 +2103,8 @@ def test_an_unresolvable_home_leaves_the_file_layer_inert(self, monkeypatch): def test_a_missing_working_directory_is_a_configuration_error(self, monkeypatch): """A job that deletes its own cwd cannot resolve a relative - DATARETRIEVAL_CONFIG. That must surface as this module's own error - type rather than a bare OSError escaping onto the request path.""" + DATARETRIEVAL_CONFIG. That must be raised as this module's own error + type rather than an unwrapped OSError propagating to the request path.""" monkeypatch.setattr( _core.Path, "cwd", @@ -2121,9 +2122,9 @@ def test_an_unreadable_config_file_names_the_path(self, tmp_path): _core._read_file_content(missing) def test_the_home_memo_watches_the_variable_that_moves_the_path(self, monkeypatch): - """``ntpath.expanduser`` ignores HOME and reads USERPROFILE, so on - Windows the memo must watch USERPROFILE or it invalidates on a - variable that cannot move the path and misses the one that can.""" + """``ntpath.expanduser`` ignores HOME and reads USERPROFILE, so on Windows the + memo must be keyed on USERPROFILE, or it invalidates on a variable that cannot + change the path and misses the one that can.""" monkeypatch.setattr(_core.os, "name", "nt") monkeypatch.setenv("USERPROFILE", r"C:\Users\ada") monkeypatch.setenv("HOME", "/ignored") @@ -2138,9 +2139,10 @@ def test_the_home_memo_watches_the_variable_that_moves_the_path(self, monkeypatc def test_show_configuration_reports_an_unresolvable_path_as_the_file_row( monkeypatch, ): - """A caller runs ``show_configuration`` precisely when their config is not - behaving. If path resolution itself fails, raising out of the explainer - withholds the one answer they came for.""" + """A caller runs ``show_configuration`` when their config is not working. If path + resolution itself fails, raising from the report withholds the information they + needed. + """ monkeypatch.setattr( configuration, "config_path", diff --git a/tests/conftest.py b/tests/conftest.py index e7a30288d..793dfb061 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -30,8 +30,9 @@ def pytest_collection_modifyitems(config, items): @pytest.fixture def non_mocked_hosts() -> list[str]: - """No hosts are exempted from mocking; every HTTP call must hit - a mock registered through the ``httpx_mock`` fixture.""" + """No hosts are exempted from mocking; every HTTP call must match a mock registered + through the ``httpx_mock`` fixture. + """ return [] @@ -43,17 +44,16 @@ def _pin_chunker_env(monkeypatch, tmp_path): ``API_USGS_RETRIES`` to 4, and ``API_USGS_STALL_TIMEOUT`` to 60 s. Pinning ``API_USGS_CONCURRENT=1`` keeps chunk dispatch deterministic for the mocked suite, and ``API_USGS_RETRIES=0`` makes - a single transient surface immediately rather than be retried. + a single transient is raised immediately rather than retried. Concurrency and retry tests opt in by overriding the env inside their body. - ``API_USGS_STALL_TIMEOUT=0`` is pinned too so that an opting-in retry - test measures the thing it names -- attempt counts -- and not the wall - clock of the machine running it. Left at the production 60 s, a test - that sets ``API_USGS_RETRIES`` would have its retries silently capped - by whatever real time its mocked attempts consumed, which is both flaky - on a loaded CI box and a way for a stall-budget bug to hide behind a - passing retry test. Tests of the budget itself set it explicitly. + ``API_USGS_STALL_TIMEOUT=0`` is pinned too so that an opting-in retry test measures + the thing it names -- attempt counts -- and not the wall clock of the machine + running it. Left at the production 60 s, a test that sets ``API_USGS_RETRIES`` would + have its retries capped, without any signal, by whatever real time its mocked + attempts consumed, which is both flaky on a busy CI machine and would let a passing + retry test conceal a stall-budget bug. Tests of the budget itself set it explicitly. """ monkeypatch.setenv("API_USGS_CONCURRENT", "1") monkeypatch.setenv("API_USGS_RETRIES", "0") diff --git a/tests/contracts/README.md b/tests/contracts/README.md index 72886e9f0..6aef9766c 100644 --- a/tests/contracts/README.md +++ b/tests/contracts/README.md @@ -3,10 +3,10 @@ The suite uses four dependency-oriented layers without moving established tests: - **Public contract** (`tests/contracts/`): imports, exports, signatures, return - annotations, metadata/error promises, and compatibility paths. These tests use + annotations, metadata/error guarantees, and compatibility paths. These tests use public modules and no live services. - **Adapter contract** (`waterdata_test.py`, `ngwmn_test.py`, `nwdc_test.py`, - `wqp_test.py`, `nldi_test.py`, `streamstats_test.py`): service request wiring, + `wqp_test.py`, `nldi_test.py`, `streamstats_test.py`): service request construction, response parsing, and documented protocol behavior. - **Component** (`transport_test.py`, `waterdata_chunking_test.py`, `waterdata_queryables_test.py`, `rdb_test.py`): one internal responsibility in diff --git a/tests/contracts/public_api_test.py b/tests/contracts/public_api_test.py index b0ed4be4a..3cee30c23 100644 --- a/tests/contracts/public_api_test.py +++ b/tests/contracts/public_api_test.py @@ -5,7 +5,7 @@ rename, reorder, and annotation reflow -- changes that break no caller -- while passing the one thing that does break callers: a new required argument, since adding one changes the text the snapshot would have to be updated to anyway. The -properties below fail on the breaking changes and stay quiet for the rest. +properties below fail on the breaking changes and pass for the rest. """ from __future__ import annotations @@ -50,7 +50,7 @@ ] #: Arguments a caller must supply positionally or by keyword. Adding an entry -#: here is a breaking change to every existing call; that is the whole reason +#: here is a breaking change to every existing call; that is why #: this mapping is written out instead of derived. _REQUIRED_ARGUMENTS = { "get_channel": (), @@ -74,10 +74,10 @@ "get_time_series_metadata": (), } -#: Defaults that are deliberately not ``None``. Every other optional parameter -#: defaults to ``None``, which is how the request builder tells "caller omitted -#: this" from "caller asked for this value" -- a non-``None`` default silently -#: adds a filter to every query. +#: Defaults that are deliberately not ``None``. Every other optional parameter defaults +#: to ``None``, which is how the request builder distinguishes an omitted argument from +#: a caller-supplied value -- a non-``None`` default adds a filter to every query +#: without the caller seeing it. _INTENTIONAL_DEFAULTS = { "convert_type": True, "expand_percentiles": True, diff --git a/tests/deprecation_test.py b/tests/deprecation_test.py index 94a0b2b53..346635a95 100644 --- a/tests/deprecation_test.py +++ b/tests/deprecation_test.py @@ -1,4 +1,5 @@ -"""The behavioural claim: downstream CI hygiene must not break the library.""" +"""The behavioural claim: a downstream ``-W error::DeprecationWarning`` +filter must not break the library.""" import warnings @@ -10,8 +11,8 @@ def test_default_wqp_calls_survive_error_on_deprecationwarning(): - """A downstream project running ``-W error::DeprecationWarning`` -- ordinary - CI hygiene -- must still be able to call wqp with default arguments. + """A downstream project running ``-W error::DeprecationWarning`` + must still be able to call wqp with default arguments. ``legacy=True`` is the default on every wqp getter and ``wqp_url`` warns unconditionally, so emitting that advisory as a ``DeprecationWarning`` @@ -24,8 +25,8 @@ def test_default_wqp_calls_survive_error_on_deprecationwarning(): def test_data_currency_is_not_a_deprecation(): - """The two categories must stay independently filterable: silencing stale - data must not silence a real removal notice, or vice versa.""" + """The two categories must stay independently filterable: filtering the stale-data + warning must not filter a removal notice, or vice versa.""" assert not issubclass(DataCurrencyWarning, DeprecationWarning) assert issubclass(DataCurrencyWarning, UserWarning) diff --git a/tests/headers_host_scoping_test.py b/tests/headers_host_scoping_test.py index 713bf39af..515435b38 100644 --- a/tests/headers_host_scoping_test.py +++ b/tests/headers_host_scoping_test.py @@ -22,41 +22,41 @@ class TestDefaultHeadersHostScoping: @pytest.fixture(autouse=True) def _api_token(self, monkeypatch: pytest.MonkeyPatch) -> None: - """Install one harmless token for every host-scoping behavior test.""" + """Set one fake token for every host-scoping behavior test.""" monkeypatch.setenv("API_USGS_PAT", self.FAKE_TOKEN) def test_key_included_for_waterdata_host(self): - """Key IS added when target URL matches api.waterdata.usgs.gov.""" + """The key is added when target URL matches api.waterdata.usgs.gov.""" url = "https://api.waterdata.usgs.gov/ogcapi/v0/collections/daily/items" headers = _default_headers(url) assert headers.get("X-Api-Key") == self.FAKE_TOKEN def test_key_excluded_for_external_host(self): - """Key is NOT added for an external (non-USGS) host.""" + """The key is not added for an external (non-USGS) host.""" url = "https://nwis.waterservices.usgs.gov/nwis/iv/" headers = _default_headers(url) assert "X-Api-Key" not in headers def test_key_excluded_for_wateruse_host(self): - """Key is NOT added for the NWDC water-use host (api.water.usgs.gov).""" + """The key is not added for the NWDC water-use host (api.water.usgs.gov).""" url = "https://api.water.usgs.gov/nwaa-data/data" headers = _default_headers(url) assert "X-Api-Key" not in headers def test_key_excluded_for_rating_asset_host(self): - """Key is NOT added for rating asset downloads (S3/external).""" + """The key is not added for rating asset downloads (S3/external).""" url = "https://labs.waterdata.usgs.gov/sta/v1.1/Datastreams(123)/rating.rdb" headers = _default_headers(url) assert "X-Api-Key" not in headers def test_key_excluded_for_lookalike_host(self): - """Key is NOT sent to a typosquatting/lookalike domain.""" + """The key is not sent to a typosquatting/lookalike domain.""" url = "https://api.waterdata.usgs.gov.evil.com/ogcapi/v0/daily/items" headers = _default_headers(url) assert "X-Api-Key" not in headers def test_key_excluded_when_no_url_provided(self): - """Key is NOT added when target_url is None (legacy callers).""" + """The key is not added when target_url is None (legacy callers).""" headers = _default_headers(None) assert "X-Api-Key" not in headers @@ -76,14 +76,14 @@ def test_non_auth_headers_always_present(self): assert "Accept" in headers assert "Accept-Encoding" in headers assert "lang" in headers - # Key should NOT be sent to example.com + # The key must not be sent to example.com assert "X-Api-Key" not in headers def test_key_excluded_over_cleartext_on_the_authorized_host(self): """The authorized host over plain http is still an unauthorized destination. - Matching on the host alone would send a bearer token in the clear on - the strength of a hostname an attacker chose to keep -- reachable via a + Matching on the host alone would send a bearer token in the clear + because of a hostname an attacker chose to keep -- reachable via a redirect or a server-supplied ``http://`` next-page link. """ headers = _default_headers("http://api.waterdata.usgs.gov/ogcapi/v0/daily") @@ -116,7 +116,7 @@ def handler(request: httpx.Request) -> httpx.Response: assert "X-Api-Key" not in seen[1].headers def test_generic_ogc_request_excludes_key_for_custom_host(self): - """A caller-supplied OGC base URL never inherits Water Data auth.""" + """A caller-supplied OGC base URL never receives Water Data auth.""" from dataretrieval.ogc.requests import _construct_api_requests request = _construct_api_requests( diff --git a/tests/ngwmn_test.py b/tests/ngwmn_test.py index 38e24cc32..d73ae0f21 100644 --- a/tests/ngwmn_test.py +++ b/tests/ngwmn_test.py @@ -1,11 +1,10 @@ """Tests for the NGWMN OGC getters (``dataretrieval.ngwmn``). -These are mocked against toy FeatureCollections shaped like the real NGWMN OGC -API (``api.waterdata.usgs.gov/ngwmn/ogcapi``) -- two features per collection, -with the real property names and value types. What is being tested is our own -request building and result shaping, and a two-row fixture exercises that just -as well as a live query does, without depending on USGS uptime or on a -particular well still having records. +These are mocked against minimal FeatureCollections shaped like the real NGWMN OGC API +(``api.waterdata.usgs.gov/ngwmn/ogcapi``) -- two features per collection, with the real +property names and value types. What is being tested is our own request building and +result shaping, and a two-row fixture exercises that as well as a live query does, +without depending on USGS uptime or on a particular well still having records. The one exception is :func:`test_state_queryables_still_diverge_upstream`, which is marked ``live``: it asserts something about the *upstream* API that a mock @@ -61,7 +60,7 @@ def _collection(features): Deliberately omits ``numberReturned``/``numberMatched``, which NGWMN does not send (the main Water Data API does) -- the pagination and shaping code - keys off ``features`` for exactly this reason, and a fixture that supplied + depends on ``features`` for this reason, and a fixture that supplied the counts would stop covering that. ``links`` is omitted too, so there is no ``next`` to follow. NGWMN does send @@ -72,10 +71,10 @@ def _collection(features): return {"type": "FeatureCollection", "features": features} -# --- toy fixtures, one per collection --------------------------------------- +# --- minimal fixtures, one per collection --------------------------------------- # Property names and value types are copied from real responses; only the number # of rows is reduced. Note the numeric-looking strings (``"4.37"``, -# ``"0"``) -- NGWMN really does send those as strings, and the dialect's +# ``"0"``) -- NGWMN sends those as strings, and the dialect's # coercion is what turns them into numbers, so the fixtures keep them as strings. _SITES = _collection( @@ -275,8 +274,9 @@ def _queries(httpx_mock, collection=None): def test_get_sites(httpx_mock): - """A sites query returns one tidy row per monitoring location, carrying - geometry by default, and reports the collection URL in its metadata.""" + """A sites query returns one row per monitoring location, with geometry by default, + and reports the collection URL in its metadata. + """ _mock(httpx_mock, "sites", _SITES) df, md = ngwmn.get_sites(state="Wisconsin", limit=10) @@ -333,7 +333,7 @@ def test_get_sites_empty_skip_geometry_is_plain(httpx_mock): def test_get_sites_state_accepts_name_postal_or_fips(httpx_mock): """The single ``state`` parameter accepts a full name, postal code, or FIPS code, and all three are normalized to the full ``state_name`` that the - ``sites`` collection actually queries on.""" + ``sites`` collection queries on.""" _mock(httpx_mock, "sites", _SITES) for encoding in ("Wisconsin", "WI", "55"): @@ -344,7 +344,7 @@ def test_get_sites_state_accepts_name_postal_or_fips(httpx_mock): for qs in sent: assert qs["state_name"] == ["Wisconsin"] # The shim rewrites into ``state_name``; raw ``state`` must not leak - # through, or the collection would silently ignore it. + # through, or the collection would ignore it without an error. assert "state" not in qs @@ -352,7 +352,7 @@ def test_get_sites_state_accepts_name_postal_or_fips(httpx_mock): def test_get_providers(httpx_mock): - """Providers carry agency/organization columns and have no geometry.""" + """Providers have agency/organization columns and have no geometry.""" _mock(httpx_mock, "providers", _PROVIDERS) df, _ = ngwmn.get_providers(state="WI") @@ -388,7 +388,7 @@ def test_get_providers_empty_stays_plain(httpx_mock): def test_get_providers_state_accepts_name_postal_or_fips(httpx_mock): """``get_providers`` normalizes any state encoding to the uppercase postal code that the ``providers`` collection queries on -- the other half of the - asymmetry that ``_STATE_QUERYABLE`` papers over.""" + asymmetry that ``_STATE_QUERYABLE`` hides.""" _mock(httpx_mock, "providers", _PROVIDERS) for encoding in ("Wisconsin", "WI", "55"): @@ -483,7 +483,7 @@ def test_get_well_construction(httpx_mock): def test_observation_collections_return_plain_dataframe(httpx_mock): - """NGWMN's observation features carry no ``geometry`` key at all (not even + """NGWMN's observation features have no ``geometry`` key at all (not even ``null``). The shaping layer has to special-case that, so assert the result is a plain frame with no geometry column rather than a GeoDataFrame.""" _mock(httpx_mock, "waterLevelObs", _WATER_LEVELS) @@ -519,8 +519,8 @@ def test_pagination_follows_next_link(httpx_mock): """Paging follows ``rel="next"`` and stops on the first page with no features. - This is the shape NGWMN actually sends: it supplies a ``next`` link even on - the last page, so an implementation that trusted the link alone would loop + This is the shape NGWMN sends: it supplies a ``next`` link even on + the last page, so an implementation that relied on the link alone would loop forever. Termination comes from the empty ``features`` array. """ page_url = ( @@ -535,7 +535,7 @@ def test_pagination_follows_next_link(httpx_mock): **_collection(_WATER_LEVELS["features"][2:]), "links": [{"rel": "next", "href": page_url, "type": "application/geo+json"}], } - # The last page carries the same ``next`` link but no features. + # The last page has the same ``next`` link but no features. last = { **_collection([]), "links": [{"rel": "next", "href": page_url, "type": "application/geo+json"}], @@ -549,8 +549,8 @@ def test_pagination_follows_next_link(httpx_mock): def test_empty_result_returns_typed_empty_frame(httpx_mock): - """A 200 carrying no features yields an empty frame whose columns come from - the collection schema, not a crash and not a shapeless frame.""" + """A 200 with no features yields an empty frame whose columns come from + the collection schema, not an exception and not a frame with no columns.""" httpx_mock.add_response( method="GET", url=_schema_re("waterLevelObs"), @@ -569,11 +569,11 @@ def test_empty_result_returns_typed_empty_frame(httpx_mock): def test_a_configured_base_url_redirects_ngwmn_alone(httpx_mock): """Two adapters share this host, and a redirect must still name only one. - NGWMN and Water Data are served from ``api.waterdata.usgs.gov``, so a URL - cannot tell them apart -- which is why the settings table an OGC call reads - is declared by the adapter rather than derived from its base. Redirecting - NGWMN therefore has to leave Water Data where it was, and the Water Data - mock here is never requested: the assertion is on the whole request list. + NGWMN and Water Data are served from ``api.waterdata.usgs.gov``, so a URL does not + distinguish them -- which is why the settings table an OGC call reads is declared by + the adapter rather than derived from its base. Redirecting NGWMN therefore has to + leave Water Data where it was, and the Water Data mock here is never requested: the + assertion is on the whole request list. """ mirror = "https://mirror.example/ngwmn" httpx_mock.add_response( @@ -602,10 +602,10 @@ def test_a_configured_base_url_redirects_ngwmn_alone(httpx_mock): @pytest.mark.live def test_state_queryables_still_diverge_upstream(): - """The NGWMN ``sites`` and ``providers`` collections expose DIFFERENT state - queryables (``sites`` -> ``state_name`` full name; ``providers`` -> - ``state`` 2-letter code). The single-``state`` shim - (``ngwmn._STATE_QUERYABLE``) exists ONLY to paper over that asymmetry. + """The NGWMN ``sites`` and ``providers`` collections expose different state + queryables (``sites`` -> ``state_name`` full name; ``providers`` -> ``state`` + 2-letter code). The single-``state`` shim (``ngwmn._STATE_QUERYABLE``) exists only + to hide that asymmetry. If this test fails, the upstream API has unified the two queryables and the shim (``_STATE_QUERYABLE``) can be removed in favor of a single pass-through diff --git a/tests/nldi_test.py b/tests/nldi_test.py index e08282881..f841afe58 100644 --- a/tests/nldi_test.py +++ b/tests/nldi_test.py @@ -59,8 +59,8 @@ def test_query_nldi_opts_into_retry(monkeypatch): monkeypatch.setattr(nldi, "_query_with_retry", query) assert nldi._query_nldi("https://example.test", {}) == {} - # ``adapter`` names whose settings the retry resolves, so a ``[nldi]`` - # table reaches these calls and no others. + # ``adapter`` names whose settings the retry resolves, so a ``[nldi]`` table applies + # to these calls and no others. query.assert_called_once_with("https://example.test", payload={}, adapter="nldi") @@ -215,7 +215,7 @@ def test_get_features_by_lat_long(httpx_mock): ], ) def test_get_features_rejects_ambiguous_origins(kwargs, problem, remedy): - """Origin validation runs ahead of the request, and names the way out. + """Origin validation runs ahead of the request, and names the remedy. Both halves are asserted because the caller is usually a program: the problem alone tells it the call is invalid, and only the remedy tells it @@ -388,7 +388,7 @@ def test_validate_data_source_rejects_invalid_after_cache_populated(httpx_mock): """Once the cache is warm, invalid data sources must still raise ValueError. Regression: previously the validation check was nested inside the - cache-population branch, so all calls after the first silently passed. + cache-population branch, so all calls after the first passed without validating. """ mock_request_data_sources(httpx_mock) @@ -402,7 +402,7 @@ def test_validate_data_source_rejects_invalid_after_cache_populated(httpx_mock): def test_search_flowlines_without_navigation_mode_raises_value_error(): - """Regression: previously crashed with AttributeError on None.upper().""" + """Regression: previously raised AttributeError on None.upper().""" with pytest.raises(ValueError, match="navigation_mode is required"): search(comid=13294314, find="flowlines") @@ -422,7 +422,7 @@ def test_search_for_basin_names_the_missing_half(kwargs, problem): """An incomplete basin origin says which argument to add, and shows one. Covers both ways the pair can be incomplete -- neither supplied, and one - of the two -- because a caller that has to guess which it hit cannot + of the two -- because a caller that has to guess which case applies cannot correct the call from the message alone. """ with pytest.raises(ValueError) as excinfo: @@ -460,7 +460,7 @@ def test_validate_navigation_mode_normalizes_lowercase(): def test_query_nldi_non_200_raises_typed_error(httpx_mock): - """A non-200 NLDI response surfaces a typed ``DataRetrievalError`` (here a + """A non-200 NLDI response raises a typed ``DataRetrievalError`` (here a 429 → ``RateLimited``, raised by the shared ``query`` path).""" from dataretrieval.exceptions import RateLimited @@ -476,10 +476,10 @@ def test_query_nldi_non_200_raises_typed_error(httpx_mock): def test_validate_data_source_rejects_malformed_catalog(httpx_mock, monkeypatch): - """``_validate_data_source`` should raise ``ValueError`` with an - informative message if the NLDI base URL returns a non-list shape - (or a list whose entries don't carry ``source`` keys), instead of - crashing with ``TypeError: string indices must be integers``.""" + """``_validate_data_source`` should raise ``ValueError`` with an informative message + if the NLDI base URL returns a non-list shape (or a list whose entries have no + ``source`` keys), instead of raising ``TypeError: string indices must be integers``. + """ monkeypatch.setattr(nldi, "_AVAILABLE_DATA_SOURCES", None) httpx_mock.add_response( method="GET", @@ -506,13 +506,13 @@ def test_query_504_raises_service_unavailable(httpx_mock): def test_a_configured_base_url_redirects_every_nldi_request(httpx_mock): - """The block moves the catalog probe and the query alike. + """The block redirects the catalog probe and the query alike. - NLDI validates a feature source against a catalog it fetches itself, so a - redirect that reached only the getter's own URL would leave the library - asking the real service whether the mirror's sources exist -- and the mirror - exists precisely because the caller cannot or should not reach the service. - Both mocks are on the mirror, so either one straying fails this. + NLDI validates a feature source against a catalog it fetches itself, so a redirect + that applied only to the getter's own URL would leave the library querying the real + service whether the mirror's sources exist -- and the mirror exists because the + caller cannot or should not contact the service. Both mocks are on the mirror, so a + request to any other host fails this. """ mirror = "https://mirror.example/nldi" httpx_mock.add_response( @@ -549,14 +549,13 @@ def test_a_configured_base_url_redirects_every_nldi_request(httpx_mock): def test_navigation_without_a_data_source_says_what_to_add(kwargs, monkeypatch): """A navigation needs the source naming which features to return. - Without this the missing source was interpolated into the path as the - literal string 'None'; the service answered 200 with an empty - FeatureCollection and the caller got an empty GeoDataFrame with no way to - tell it apart from a navigation that really has nothing on it. + Without this the missing source was interpolated into the path as the literal string + 'None'; the service returned 200 with an empty FeatureCollection, and the caller + received an empty GeoDataFrame indistinguishable from a navigation with no features. """ - # Seed the catalog: the feature_source case validates it on the way past, - # and the autouse fixture clears it, so an unseeded run reaches the network - # for a failure that is purely local. + # Seed the catalog: the feature_source case validates it before failing, and the + # autouse fixture clears it, so an unseeded run would make a network request for a + # failure that is local. monkeypatch.setattr(nldi, "_AVAILABLE_DATA_SOURCES", ["WQP", "nwissite"]) with pytest.raises(ValueError) as excinfo: get_features(**kwargs) @@ -568,9 +567,9 @@ def test_navigation_without_a_data_source_says_what_to_add(kwargs, monkeypatch): def test_a_bad_navigation_mode_is_reported_before_the_missing_data_source(): """Both arguments are invalid; the mode is the one the caller typed. - Requiring ``data_source`` ahead of validating the mode would answer a - mistyped ``navigation_mode`` with a message about a different argument, - so the caller fixes that, re-runs, and only then learns about the typo. + Requiring ``data_source`` ahead of validating the mode would respond to a mistyped + ``navigation_mode`` with a message about a different argument, so the caller fixes + that, re-runs, and only then sees the typo. """ with pytest.raises(ValueError) as excinfo: get_features(comid=13294314, navigation_mode="XX") @@ -599,13 +598,13 @@ def test_get_features_by_data_source_validates_the_source(httpx_mock): def test_a_200_with_a_non_json_body_becomes_an_empty_frame(httpx_mock): - """NLDI answers some queries 200 with an empty body, and that is not an - error condition -- a feature with nothing upstream is a real answer. + """NLDI returns 200 with an empty body for some queries, and that is not an + error condition -- a feature with nothing upstream is a valid result. - This is the one place the package returns an empty frame rather than - raising on a malformed response. Pinned because it is deliberate: the - swallow reads as an oversight and could be 'fixed' into a raise, which - would turn a legitimate empty navigation into a crash. + This is the one place the package returns an empty frame rather than raising on a + malformed response. Pinned because it is deliberate: the suppression reads as an + oversight and could be 'fixed' into a raise, which would make a legitimate empty + navigation raise. """ mock_request_data_sources(httpx_mock) httpx_mock.add_response( @@ -619,11 +618,11 @@ def test_a_200_with_a_non_json_body_becomes_an_empty_frame(httpx_mock): assert isinstance(gdf, GeoDataFrame) assert gdf.empty - assert gdf.crs is not None # the CRS survives the empty path + assert gdf.crs is not None # the CRS is kept on the empty path def test_get_flowlines_forwards_stop_comid(httpx_mock): - """``stop_comid`` bounds a navigation and must reach the query string.""" + """``stop_comid`` bounds a navigation and must be included in the query string.""" request_url = ( f"{NLDI_API_BASE_URL}/comid/13294314/navigation/UM/flowlines" "?distance=50&trimStart=false&stopComid=13294312" @@ -642,7 +641,7 @@ def test_get_flowlines_forwards_stop_comid(httpx_mock): def test_search_rejects_a_basin_lookup_by_comid(): """A basin is looked up by feature, not by flowline; the message must - offer both ways forward rather than only naming the conflict.""" + offer both remedies rather than only naming the conflict.""" with pytest.raises(ValueError) as excinfo: search(find="basin", comid=13294314) message = str(excinfo.value) diff --git a/tests/nwdc_test.py b/tests/nwdc_test.py index a089833ef..d41fbb904 100644 --- a/tests/nwdc_test.py +++ b/tests/nwdc_test.py @@ -25,8 +25,8 @@ from dataretrieval.transport import fanout as _fanout from dataretrieval.utils import BaseMetadata -# Match the NWDC endpoint regardless of query string, so assertions can drill -# into the captured params without coupling registration to param order. +# Match the NWDC endpoint regardless of query string, so assertions can inspect the +# captured params without coupling registration to param order. WU_RE = re.compile(r"^https://api\.water\.usgs\.gov/nwaa-data/data(\?.*)?$") # A single-page monthly CSV: two HUC12s (one with a leading zero), three months. @@ -80,7 +80,7 @@ def test_huc12_id_kept_as_string_with_leading_zero(httpx_mock): df, _ = get_wateruse(model="wu-public-supply-wd", state="RI") # String-typed (object or the pandas StringDtype, depending on version), - # never coerced to int — the leading zero must survive. + # never coerced to int — the leading zero must be kept. assert pd.api.types.is_string_dtype(df["huc12_id"]) assert df["huc12_id"].iloc[0] == "010900020502" @@ -161,7 +161,7 @@ def test_pagination_follows_link_header_and_concatenates(httpx_mock): ] assert list(df.index) == [0, 1, 2] assert len(httpx_mock.get_requests()) == 2 - # The second request carries the Link's ``skip`` offset, not the originals. + # The second request includes the Link's ``skip`` offset, not the originals. second_qs = parse_qs(urlsplit(str(httpx_mock.get_requests()[1].url)).query) assert second_qs["skip"] == ["2"] @@ -188,7 +188,7 @@ def test_pagination_rewrites_bare_host(httpx_mock): def test_http_error_raises_typed_exception_with_detail(httpx_mock): - """A 4xx response surfaces as a typed error carrying the NWDC ``detail``.""" + """A 4xx response is raised as a typed error that includes the NWDC ``detail``.""" httpx_mock.add_response( method="GET", url=WU_RE, @@ -210,7 +210,7 @@ def test_empty_response_body_raises_typed_error(httpx_mock): def test_cyclic_next_link_terminates(httpx_mock): """A non-advancing/cyclic ``next`` cursor must not loop forever.""" - # Page 1 points to a "next" URL; page 2 points back to that SAME URL. + # Page 1 points to a "next" URL; page 2 points back to the same URL. cyclic = ( "; rel="next"' @@ -231,7 +231,7 @@ def test_cyclic_next_link_terminates(httpx_mock): def test_uses_shared_default_headers(httpx_mock): - """Requests carry the shared dataretrieval User-Agent (per _default_headers).""" + """Requests include the shared dataretrieval User-Agent (per _default_headers).""" httpx_mock.add_response(method="GET", url=WU_RE, text=_CSV_PAGE) get_wateruse(model="wu-public-supply-wd", state="RI") @@ -325,7 +325,7 @@ def test_fan_out_failure_never_returns_partial_data(httpx_mock): """A failed location aborts the call even when another location succeeded. The completed sibling is not returned as though the call had succeeded -- - it is carried on the raised interruption for ``resume()`` instead. Water Use + it is held on the raised interruption for ``resume()`` instead. Water Use reports ``ServiceInterrupted`` rather than the bare ``ServiceUnavailable`` it raised before sharing the fan-out executor: the same upstream 503, now resumable. @@ -345,8 +345,8 @@ def test_fan_out_failure_never_returns_partial_data(httpx_mock): with pytest.raises(dataretrieval.ServiceInterrupted) as excinfo: get_wateruse(model="wu-public-supply-wd", state=["RI", "WI"]) - # The 503 is still the reported cause, and the successful location survives - # on the exception rather than being passed off as the whole answer. + # The 503 is still the reported cause, and the successful location is kept on the + # exception rather than returned as though it were the whole result. assert isinstance(excinfo.value.__cause__, dataretrieval.ServiceUnavailable) assert excinfo.value.status_code == 503 assert excinfo.value.retryable @@ -359,7 +359,7 @@ def test_fan_out_failure_never_returns_partial_data(httpx_mock): def test_resolve_locations_state_accepts_name_postal_fips(): - # All three encodings normalize to the two-letter postal code stateCd wants. + # All three encodings normalize to the two-letter postal code stateCd accepts. assert _resolve_locations("Rhode Island", None, None) == ["stateCd:RI"] assert _resolve_locations("ri", None, None) == ["stateCd:RI"] assert _resolve_locations("44", None, None) == ["stateCd:RI"] @@ -482,7 +482,7 @@ def test_next_page_url_normalizes_other_spellings_of_the_same_service(): """The cursor is normalized by host, not by one literal prefix. A plain-http or relative ``next`` link is the same service; refusing it - would throw away every page already collected for that location. + would discard every page already collected for that location. """ plain_http = httpx.Response( 200, @@ -507,11 +507,11 @@ def test_next_page_url_normalizes_other_spellings_of_the_same_service(): def test_next_page_url_strips_credentials_from_the_cursor(): """Userinfo on a cursor must not become an Authorization header. - httpx derives ``Authorization: Basic ...`` from a URL's userinfo, so a - cursor spelled ``http://user:pass@water.usgs.gov/...`` would send a - credential the caller never configured to the rewritten host -- exactly what - the host check exists to prevent, arriving through the host check's own - normalization. The port is dropped for the same reason. + httpx derives ``Authorization: Basic ...`` from a URL's userinfo, so a cursor + spelled ``http://user:pass@water.usgs.gov/...`` would send a credential the caller + never configured to the rewritten host -- what the host check exists to prevent, + reached through the host check's own normalization. The port is dropped for the same + reason. """ response = httpx.Response( 200, @@ -530,7 +530,7 @@ def test_next_page_url_strips_credentials_from_the_cursor(): assert "s3cret" not in cursor assert httpx.URL(cursor).userinfo == b"" - # Assert at the layer that actually synthesizes the header: ``httpx.Request`` + # Assert at the layer that synthesizes the header: ``httpx.Request`` # never derives Basic auth from userinfo (so asserting there would pass for # any URL) -- the ``Client`` does it at send time. sent: dict[str, str | None] = {} @@ -545,12 +545,12 @@ def capture(request: httpx.Request) -> httpx.Response: def test_a_configured_base_url_redirects_the_request(httpx_mock): - """The whole call moves, page walk included, or the redirect is a half-truth. + """The whole call moves, page walk included, or the redirect is incomplete. - The page-two mock is served from the mirror and its cursor names the mirror: - if either the request or the ``rel="next"`` walk had stayed on the NWDC's - host, one of them would go unmocked and this would fail rather than quietly - talk to the service the block redirected away from. + The page-two mock is served from the mirror and its cursor names the mirror: if + either the request or the ``rel="next"`` walk had stayed on the NWDC's host, one of + them would go unmocked and this would fail rather than send requests, undetected, to + the service the block redirected away from. """ mirror = re.compile(r"^https://mirror\.example/data") httpx_mock.add_response( @@ -576,11 +576,11 @@ def test_a_configured_base_url_redirects_the_request(httpx_mock): def test_next_page_url_drops_the_service_rewrite_when_redirected(): """The alias list and the rewrite are facts about the NWDC, not about URLs. - Nothing but the NWDC answers for ``water.usgs.gov``, so a call an + Nothing but the NWDC is served at ``water.usgs.gov``, so a call an ``NwdcConfiguration(base_url=...)`` pointed elsewhere gets the general rule instead: follow a link only back to the host that served the page. Keeping the rewrite would send page two of a mirrored query to the USGS -- and - refusing the mirror's own cursor would throw away page one. + refusing the mirror's own cursor would discard page one. """ mirrored = httpx.Response( 200, @@ -628,14 +628,14 @@ def test_initial_transient_is_retried(httpx_mock, monkeypatch): def test_fatal_failure_waits_for_siblings_before_closing_the_client(monkeypatch): - """A fan-out failure must not close the client under its own siblings. - - Every location shares one ``httpx.AsyncClient`` scoped to the fan-out. When - the first failure propagated straight out of the ``gather``, that block - exited while siblings were still walking pages, and the next page they asked - for failed with "Cannot send a request, as the client has been closed" -- on - a task nobody was awaiting any more, so it also surfaced as an unretrieved - exception. Both are artifacts of our own teardown, not of the service. + """A fan-out failure must not close the client while its siblings still use it. + + Every location shares one ``httpx.AsyncClient`` scoped to the fan-out. When the + first failure propagated straight out of the ``gather``, that block exited while + siblings were still walking pages, and the next page they requested failed with + "Cannot send a request, as the client has been closed" -- on a task nobody was + awaiting any more, so it was also reported as an unretrieved exception. Both are + caused by our own teardown, not by the service. """ import asyncio from contextlib import asynccontextmanager @@ -681,7 +681,7 @@ async def open_mock_client(**overrides): with pytest.raises(dataretrieval.DataRetrievalError, match="Invalid model"): nwdc._fan_out(requests, {}, True) - assert pages["n"] == 2, "the sibling finished its walk rather than being abandoned" + assert pages["n"] == 2, "the sibling finished its walk rather than being cancelled" def test_next_page_url_rejects_cross_host_link(): @@ -701,10 +701,9 @@ def test_next_page_url_rejects_cross_host_link(): def test_interrupted_fan_out_resumes_only_the_unfinished_locations(httpx_mock): """A rate-limited location is resumable; completed siblings are not re-fetched. - Before Water Use shared the executor, a 429 anywhere in the fan-out - discarded every location that had already succeeded. That is the whole - reason a multi-location pull needed re-running from scratch against an - hourly quota. + Before Water Use shared the executor, a 429 anywhere in the fan-out discarded every + location that had already succeeded. That is why a multi-location pull had to be + re-run in full against an hourly quota. """ httpx_mock.add_response( method="GET", url=re.compile(r".*location=stateCd%3ARI.*"), text=_CSV_P1 @@ -733,7 +732,7 @@ def test_interrupted_fan_out_resumes_only_the_unfinished_locations(httpx_mock): df, md = interrupted.call.resume() - # Only WI was re-issued; RI's completed frame carried across the resume. + # Only WI was re-issued; RI's completed frame was kept across the resume. assert len(httpx_mock.get_requests()) == requests_before + 1 assert len(df) == 3 assert isinstance(md, BaseMetadata) @@ -742,8 +741,8 @@ def test_interrupted_fan_out_resumes_only_the_unfinished_locations(httpx_mock): def test_fan_out_honors_the_general_concurrency_setting(monkeypatch): """``API_USGS_CONCURRENT`` outranks this service's default. - A user dialing concurrency down to be polite must not find Water Use - quietly ignoring them -- the defect that motivated consolidating the knob. + A user lowering concurrency to reduce load must not find Water Use + ignoring the setting -- the defect that motivated sharing one setting. """ monkeypatch.setenv("API_USGS_CONCURRENT", "7") assert configuration.concurrency(nwdc.DEFAULT_CONCURRENT_REQUESTS) == 7 @@ -758,7 +757,9 @@ def test_fan_out_honors_the_general_concurrency_setting(monkeypatch): def test_fan_out_reports_progress(httpx_mock, monkeypatch): - """The fan-out ticks the progress reporter, which it never did standalone.""" + """The fan-out updates the progress reporter, which it did not do before sharing the + executor. + """ seen = [] class _Recorder: @@ -789,7 +790,7 @@ def add_page(self, rows): def test_resume_uses_the_current_progress_reporter(httpx_mock, monkeypatch): - """Resume must not resurrect the reporter closed by the interrupted call.""" + """Resume must not reuse the reporter closed by the interrupted call.""" created = [] class _Recorder: @@ -891,12 +892,12 @@ def test_permanent_later_page_failure_remains_a_network_error(httpx_mock): def test_mid_page_walk_transient_is_still_resumable(httpx_mock): """A 429 on page 2+ of a location must still be a resumable interruption. - ``paginate`` re-wraps a later-page failure as a plain ``DataRetrievalError`` - (page 1's status check sits outside its ``try``), so the typed cause is only - reachable through ``__cause__``. ``_classify_chunk_error`` walks that chain - for exactly this reason; were it a single ``isinstance`` check, a mid-walk - rate limit would escape as a bare error and lose ``.call.resume()`` -- - inconsistently, since page 1 would still be resumable. + ``paginate`` re-wraps a later-page failure as a plain ``DataRetrievalError`` (page + 1's status check is outside its ``try``), so the typed cause is only reachable + through ``__cause__``. ``_classify_chunk_error`` walks that chain for this reason; + were it a single ``isinstance`` check, a mid-walk rate limit would propagate as a + bare error and lose ``.call.resume()`` -- inconsistently, since page 1 would still + be resumable. """ httpx_mock.add_response( method="GET", @@ -930,7 +931,7 @@ def test_mid_page_walk_transient_is_still_resumable(httpx_mock): def _reimport_wateruse(): - """Import the alias fresh, so its module-level warning fires again.""" + """Re-import the alias, so its module-level warning is emitted again.""" import importlib import sys @@ -956,7 +957,7 @@ def test_wateruse_alias_warns_and_names_the_replacement(): def test_wateruse_alias_re_exports_the_same_objects(): - """The alias forwards, it does not copy: identity must survive it. + """The alias forwards rather than copies, so identity is preserved through it. A caller monkeypatching through one spelling and asserting through the other would otherwise see two different objects. @@ -971,15 +972,15 @@ def test_wateruse_alias_re_exports_the_same_objects(): def test_importing_dataretrieval_does_not_warn(): - """``import dataretrieval`` must stay silent. + """``import dataretrieval`` must emit no warning. The package imports ``nwdc`` directly; only code naming ``wateruse`` itself should see the warning. If ``__init__`` ever imports the alias, every user of the library gets a DeprecationWarning they cannot act on. - Runs in a subprocess: a fresh interpreter is the only way to observe - an import side effect, and clearing ``sys.modules`` in-process would hand - every later test a second copy of the package. + Runs in a subprocess: a new interpreter is the only way to observe an import side + effect, and clearing ``sys.modules`` in-process would give every later test a second + copy of the package. """ import subprocess import sys diff --git a/tests/nwis_test.py b/tests/nwis_test.py index c70b6e5b3..92231e18b 100644 --- a/tests/nwis_test.py +++ b/tests/nwis_test.py @@ -59,7 +59,7 @@ def _test_iv_service(httpx_mock): service = "iv" site = ["03339000", "05447500", "03346500"] - # We use a very simple JSON structure just to satisfy the parser + # A minimal JSON structure to satisfy the parser mock_json = _load_mock_json("nwis_iv_mock.json") # Match the base URL and ensure query parameters are correct @@ -92,14 +92,13 @@ def test_iv_service_answer(httpx_mock): ], ) def test_preformat_peaks_response_keeps_every_peak(peak_dt, expected): - """A peak is never dropped for want of a parseable date. - - NWIS zero-fills the unknown part of a historical peak's date -- - ``YYYY-MM-00`` when the day is not known, ``YYYY-00-00`` when the month is - not either (the ``Bd`` and ``Bm`` ``peak_cd`` qualifiers). Those are real - peaks, often a site's largest, and dropping them loses the discharge value - with the date. A date NWIS only partly knows stays ``NaT`` rather than - being completed into one it does not have. + """A peak is never dropped because its date cannot be parsed. + + NWIS zero-fills the unknown part of a historical peak's date -- ``YYYY-MM-00`` when + the day is not known, ``YYYY-00-00`` when the month is not either (the ``Bd`` and + ``Bm`` ``peak_cd`` qualifiers). Those are real peaks, often a site's largest, and + dropping them loses the discharge value with the date. A date NWIS records only + partly stays ``NaT`` rather than being completed into one it does not record. """ df = pd.DataFrame({"peak_dt": [peak_dt], "peak_va": [563000]}) @@ -114,12 +113,12 @@ def test_preformat_peaks_response_keeps_every_peak(peak_dt, expected): def test_preformat_peaks_response_preserves_peak_dt(): - """``peak_dt`` must survive the reformat. + """``peak_dt`` must be kept through the reformat. - The peaks response carries no ``water_yr``, so ``peak_dt`` is the only + The peaks response has no ``water_yr``, so ``peak_dt`` is the only column holding the year of a censored peak -- and the only way a caller can tell an unknown day from a known one, since ``peak_cd`` does not always - carry the qualifier. + include the qualifier. """ df = pd.DataFrame({"peak_dt": ["1858-00-00"], "peak_va": [563000]}) @@ -131,7 +130,7 @@ def test_preformat_peaks_response_preserves_peak_dt(): def test_preformat_peaks_response_malformed_frame_still_raises(): """Only an *empty* peaks frame is a legitimate empty result. A non-empty frame with no ``peak_dt`` column is a malformed response -- a truncated or - altered RDB header -- and must stay loud rather than be returned silently + altered RDB header -- and must raise rather than be returned without its datetime index. """ df = pd.DataFrame({"peak_va": [1000]}) @@ -141,9 +140,9 @@ def test_preformat_peaks_response_malformed_frame_still_raises(): class TestDeprecationWarnings: - """Verify per-function DeprecationWarning fires with the right replacement. + """Verify the per-function DeprecationWarning is emitted with the right replacement. - The module-level "use waterdata instead" warning fires on import; these + The module-level "use waterdata instead" warning is emitted on import; these tests pin the function-specific replacements so users see actionable migration guidance the first time they call each NWIS getter. """ @@ -214,11 +213,11 @@ def test_nested_calls_emit_one_warning(self, httpx_mock): ], ) def test_named_replacement_exists_in_waterdata(self, name): - """Tripwire: every concrete `waterdata.*` named in a deprecation message - must actually exist, so a user following the migration guidance doesn't - hit AttributeError. + """Every concrete `waterdata.*` named in a deprecation message + must exist, so a user following the migration guidance does not + get an AttributeError. - Fails loudly if this PR ever lands before its referenced replacement + Fails if this change is ever merged before its referenced replacement does (e.g. before `get_peaks` from #267). """ import dataretrieval.waterdata as wd @@ -336,7 +335,7 @@ class TestReadRdb: The format-agnostic parser is exercised in tests/rdb_test.py; this class pins the wrapper-specific contract — that an empty parser - result flows through format_response without crashing (issue #171), + result passes through format_response without raising (issue #171), on the plain arm and on the peaks arm alike. """ @@ -367,7 +366,7 @@ def test_no_peaks_flows_through_format_response(self): Both functions are public API, so any caller parsing a peaks RDB reaches this -- it is not unreachable behind ``NoSitesError``. """ - # Mirror get_discharge_peaks: raw read_rdb, then the peaks-specific + # Match get_discharge_peaks: raw read_rdb, then the peaks-specific # format_response. df = read_rdb(self.NO_RESULTS_RDB) df = format_response(df, service="peaks") @@ -377,8 +376,8 @@ def test_no_peaks_flows_through_format_response(self): def test_malformed_peaks_frame_still_raises(self): """Only an *empty* peaks frame is a legitimate empty result. A non-empty frame with no ``peak_dt`` column is a malformed response -- - a truncated or altered RDB header -- and must stay loud rather than be - returned silently without its datetime index. + a truncated or altered RDB header -- and must raise rather than be + returned without its datetime index. """ df = pd.DataFrame({"peak_va": [1000]}) @@ -389,10 +388,10 @@ def test_malformed_peaks_frame_still_raises(self): class TestGetRecordDispatch: """``get_record`` is a router; each service must reach its own getter. - The arms are near-identical by eye, which is what makes a mis-wired one - survive review: every arm forwards ``sites`` except ``ratings``, which - takes a scalar ``site``. A swap there fails only at request time, for one - service, in a deprecated facade nobody reads. + The arms look near-identical, which is what lets a wrong one pass review: every arm + forwards ``sites`` except ``ratings``, which takes a scalar ``site``. A swap there + fails only at request time, for one service, in a deprecated facade that is rarely + read. """ @pytest.mark.parametrize( @@ -426,11 +425,11 @@ def test_unrecognized_service_lists_the_ones_it_serves(self): def test_html_error_page_instead_of_json_says_what_to_do(): - """A 200 carrying an HTML error page must not surface as a JSON parse error. + """A 200 whose body is an HTML error page must not be raised as a JSON parse error. - The legacy services answer an outage with a styled page and a 200, so the - only signal is the body. A caller that gets ``JSONDecodeError`` learns - nothing actionable; this path names the cause and the move. + The legacy services respond to an outage with a styled page and a 200, so the + only signal is the body. A caller that gets ``JSONDecodeError`` has + nothing actionable; this path names the cause and the remedy. """ response = mock.Mock() response.json.side_effect = ValueError("no json") @@ -448,7 +447,7 @@ def test_html_error_page_instead_of_json_says_what_to_do(): def test_a_non_html_parse_failure_is_re_raised_unchanged(): - """Only HTML gets the rewrite; a genuine malformed-JSON body must not be + """Only HTML gets the rewrite; a malformed-JSON body must not be relabelled as a service outage.""" response = mock.Mock() response.json.side_effect = ValueError("Expecting value") @@ -462,9 +461,9 @@ def test_a_non_html_parse_failure_is_re_raised_unchanged(): def test_deprecating_a_getter_with_no_named_replacement_is_refused(): - """``@_deprecated`` promises the caller a replacement, so the decorator - refuses to be applied to a function whose replacement nobody recorded -- - a deprecation warning naming nothing leaves the caller with nothing to + """``@_deprecated`` guarantees the caller a replacement, so applying it to a + function with no recorded replacement raises -- a deprecation warning naming nothing + leaves the caller with nothing to migrate to.""" with pytest.raises(RuntimeError, match="_REPLACEMENTS missing entry"): @@ -494,9 +493,9 @@ def test_metadata_site_info_is_none_when_no_site_filter_was_used(): def test_utc_localization_of_a_multi_index_datetime_level(): - """``multi_index=True`` puts the timestamp on level 1 under the site id. - The naive level must still be localized, or a multi-site frame carries - two different clock conventions in one column.""" + """``multi_index=True`` puts the timestamp on level 1 under the site id. The naive + level must still be localized, or a multi-site frame holds two different time-zone + conventions in one column.""" idx = pd.MultiIndex.from_arrays( [ ["01491000", "01491000"], @@ -509,9 +508,9 @@ def test_utc_localization_of_a_multi_index_datetime_level(): class TestGetInfoSeriesCatalog: - """``seriesCatalogOutput`` and the expanded site format are mutually - exclusive on the wire, so the getter picks one and warns when the caller - asked for the retiring one.""" + """``seriesCatalogOutput`` and the expanded site format are mutually exclusive on + the wire, so the getter picks one and warns when the caller asked for the one being + retired.""" @pytest.mark.parametrize("flag", ["True", "TRUE", "true", True]) def test_asking_for_the_series_catalog_warns_and_forwards_it( diff --git a/tests/rdb_test.py b/tests/rdb_test.py index 99f46ff55..0625b626e 100644 --- a/tests/rdb_test.py +++ b/tests/rdb_test.py @@ -49,7 +49,7 @@ def test_read_rdb_empty_when_only_comments(): def test_read_rdb_raises_on_html_response(): - """If the service returns an HTML error page, surface it loudly.""" + """If the service returns an HTML error page, raise.""" with pytest.raises(ValueError, match="HTML"): read_rdb("Service Unavailable") with pytest.raises(ValueError, match="HTML"): diff --git a/tests/streamstats_test.py b/tests/streamstats_test.py index 8b356c94d..76f355398 100644 --- a/tests/streamstats_test.py +++ b/tests/streamstats_test.py @@ -22,7 +22,7 @@ def test_watershed_from_streamstats_json_builds_independent_instances(): """B3 regression: ``from_streamstats_json`` previously wrote *class* attributes and returned the class object, so it produced no real - instance and a second parse clobbered the first. It must now return + instance and a second parse overwrote the first. It must now return an independent, populated ``Watershed`` instance.""" w1 = Watershed.from_streamstats_json(_SAMPLE) assert isinstance(w1, Watershed) # was the class object pre-fix @@ -33,7 +33,7 @@ def test_watershed_from_streamstats_json_builds_independent_instances(): w2 = Watershed.from_streamstats_json(dict(_SAMPLE, workspaceID="WS-XYZ")) assert w1 is not w2 - assert w1._workspaceID == "WS-ABC" # not clobbered by w2 (was shared class state) + assert w1._workspaceID == "WS-ABC" # not overwritten by w2 (was shared class state) assert w2._workspaceID == "WS-XYZ" @@ -56,8 +56,8 @@ def test_get_watershed_geojson_returns_raw_response(httpx_mock): def test_get_watershed_shape_raises_not_implemented(httpx_mock): - """B3: the unimplemented ``format='shape'`` must fail loudly rather - than silently falling through to a (previously broken) ``Watershed``.""" + """B3: the unimplemented ``format='shape'`` must raise rather + than fall through to a (previously broken) ``Watershed``.""" httpx_mock.add_response(text=json.dumps(_SAMPLE)) with pytest.raises(NotImplementedError): get_watershed("NY", -74.524, 43.939, format="shape") @@ -96,8 +96,8 @@ def test_get_watershed_retries_transient_failure(httpx_mock, monkeypatch): def test_watershed_constructor_delineates_and_parses(monkeypatch): - """``Watershed(...)`` is the object-shaped entry point: it must issue the - geojson request and land the parsed fields on the instance, so a caller + """``Watershed(...)`` is the entry point that returns an object: it must issue the + geojson request and set the parsed fields on the instance, so a caller never handles the raw response.""" import httpx @@ -120,7 +120,7 @@ def fake_get_watershed(rcode, x, y, **kwargs): def test_get_sample_watershed_uses_the_documented_sample_location(monkeypatch): """The sample helper exists so a new user can get a real object in one - call; it must keep asking for the location the docstring advertises.""" + call; it must keep requesting the location the docstring documents.""" from dataretrieval import streamstats captured = {} diff --git a/tests/transport_test.py b/tests/transport_test.py index e2c0f3968..72b090147 100644 --- a/tests/transport_test.py +++ b/tests/transport_test.py @@ -168,7 +168,7 @@ async def operation(item: str) -> tuple[pd.DataFrame, httpx.Response]: def test_retry_tunables_have_a_single_home() -> None: - """Patching retry tunables must reach the policy that reads them.""" + """Patching retry tunables must apply to the policy that reads them.""" from dataretrieval import interruptions assert not [name for name in vars(interruptions) if name.startswith("_RETRY")] @@ -183,11 +183,11 @@ def test_retired_ogc_retry_shim_stays_absent() -> None: def test_parse_retry_after_accepts_http_date() -> None: - """A date in the future is honored; one already past is not a hint. + """A date in the future is applied; one already past is not a value. Read literally an elapsed date says "retry now", but the likelier cause is our clock running ahead of the server's, and acting on it would re-send - almost immediately against a service that just asked for a pause. + almost immediately against a service that just named a delay. """ soon = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(seconds=30) parsed = exceptions.parse_retry_after(soon.strftime("%a, %d %b %Y %H:%M:%S GMT")) @@ -202,8 +202,8 @@ def test_parse_retry_after_accepts_http_date() -> None: def test_both_retry_after_forms_are_honored_alike() -> None: """The two header spellings mean the same thing and must behave the same. - Discarding an over-long date hint (returning ``None``) made the client retry - *harder* against a service asking for a long pause, and dropped the number + Discarding an over-long date value (returning ``None``) made the client retry + sooner against a service that named a long delay, and dropped the number the caller needs from ``.retry_after``. """ far_future = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta( @@ -225,22 +225,22 @@ def test_elapsed_retry_after_still_backs_off() -> None: policy = retry.RetryPolicy(base_backoff=0.5, max_backoff=30.0) assert policy.backoff(attempt=1, retry_after=0.0) > 0.0 - # The nudge is bounded by max_backoff, not this attempt's exponential - # ceiling: keying it to the ceiling made it vanish whenever base_backoff was - # zero -- exactly when a hint of 0 would become a zero-delay re-send. + # The jitter is bounded by max_backoff, not this attempt's exponential ceiling: + # keying it to the ceiling made it zero whenever base_backoff was zero -- when a + # value of 0 would become a zero-delay re-send. assert retry.RetryPolicy(base_backoff=0.0).backoff(attempt=1, retry_after=0.0) > 0.0 - # A server-named delay is honored, plus a small decorrelating nudge so - # concurrent chunks handed the same hint do not all wake together -- + # A server-named delay is applied, plus a small decorrelating offset so + # concurrent chunks given the same value do not all retry together -- # and never enough to push the wait past the policy's own bounds. assert 5.0 < policy.backoff(attempt=1, retry_after=5.0) <= 6.0 - # A hint already at the cap is never nudged past it -- the jitter would + # A value already at the cap is never extended past it -- the jitter would # otherwise sleep longer than any bound the policy declares. at_cap = policy.backoff(attempt=8, retry_after=policy.retry_after_cap) assert at_cap == policy.retry_after_cap def _dns_failure(errno: int) -> NetworkError: - """A DNS failure shaped the way one actually reaches the retry loop. + """A DNS failure shaped the way one reaches the retry loop. httpx and httpcore link their wrappers with ``__context__`` (implicit chaining), not ``__cause__``, so a walker following only explicit causes @@ -255,7 +255,7 @@ def _dns_failure(errno: int) -> NetworkError: def test_deterministic_failures_are_not_retried() -> None: - """Only failures a later attempt could survive are worth re-sending. + """Only failures a later attempt could succeed on are re-sent. The ``EAI_*`` values are platform-specific -- ``EAI_NONAME`` is 8 on macOS and -2 on Linux -- so these must come from :mod:`socket` rather @@ -270,12 +270,12 @@ def test_deterministic_failures_are_not_retried() -> None: def test_temporary_name_resolution_is_still_retried() -> None: """``gaierror`` is not one condition: ``EAI_AGAIN`` means "try again". - A resolver still coming up, a VPN reconnect, or a laptop waking all - surface this way, and they are exactly the failures retry exists for. + A resolver still starting, a VPN reconnect, or a laptop resuming from + sleep all produce this code, and they are the failures retry exists for. """ assert retry._retryable(_dns_failure(socket.EAI_AGAIN)) == (True, None) - # An unrecognized code is retried too: a wasted attempt is cheaper than - # dropping a call we could have recovered. + # An unrecognized code is retried too: one wasted attempt costs less than + # abandoning a recoverable call. assert retry._retryable(_dns_failure(0)) == (True, None) @@ -284,7 +284,7 @@ def test_resolver_failure_found_past_an_unrelated_explicit_cause() -> None: ``raise X from Y`` inside an ``except`` block leaves an explicit ``__cause__`` *and* an unrelated ``__context__`` on the same frame. Following - only the cause walks off down the explicit branch and never reaches the + only the cause takes the explicit branch and never reaches the ``gaierror``, so an unresolvable hostname spends the whole retry budget instead of failing fast. """ @@ -310,20 +310,20 @@ def test_chain_walk_terminates_on_a_self_referential_cause() -> None: def test_retryable_statuses_are_per_adapter() -> None: """A 500 means different things to different services, so the set differs. - WQP answers an over-large query with a 500 and StreamStats answers - out-of-network coordinates with one, so re-sending can never help there. The - Water Data OGC API is a query interface where a 500 is an upstream hiccup, so - the chunker keeps riding those out — applying WQP's rationale to it would - quietly drop retries the chunked getters have always had. + WQP responds to an over-large query with a 500 and StreamStats responds to + out-of-network coordinates with one, so re-sending can never help there. The Water + Data OGC API is a query interface where a 500 is a transient upstream failure, so + the chunker keeps retrying those — applying WQP's rationale to it would drop retries + the chunked getters have always had. """ rejected_query = ServiceUnavailable("bad query", status_code=500) gateway = ServiceUnavailable("bad gateway", status_code=502) - # Default (Water Data chunker): every 5xx is worth another try. + # Default (Water Data chunker): every 5xx is retried. assert retry._retryable(rejected_query)[0] assert retry._retryable(gateway)[0] - # One-shot adapters: only the gateway family. + # One-shot adapters: only the gateway statuses. strict = retry._GATEWAY_STATUSES assert not retry._retryable(rejected_query, strict)[0] assert retry._retryable(gateway, strict)[0] @@ -336,11 +336,12 @@ def test_retryable_statuses_are_per_adapter() -> None: def test_stall_timeout_stops_a_silent_call(monkeypatch) -> None: - """Retrying stops once a call has gone quiet for the whole budget. + """Retrying stops once a call has received nothing for the whole budget. Without this, a request that times out is retried until the attempts run out, turning one 60 s timeout into minutes of apparent hang. The first - retry is exempt (see below), so a silent call costs two attempts, not five. + retry is exempt (see below), so a call that receives nothing costs two + attempts, not five. """ attempts = 0 @@ -363,13 +364,13 @@ def operation() -> str: def test_server_named_delay_does_not_consume_the_stall_budget(monkeypatch) -> None: - """Honoring ``Retry-After`` must not cost a call its retries. + """Applying ``Retry-After`` must not cost a call its retries. - The budget bounds *silence*; a delay the service named is the opposite of - going quiet. Charging for it meant the more politely a service asked for - room, the fewer retries it got: with the shipped defaults a + The budget bounds time without data; a delay the service named is not + that. Charging for it meant the longer a service asked the client to + wait, the fewer retries the call got: with the shipped defaults a ``Retry-After: 30`` against a 60 s budget allowed exactly one retry no - matter what ``API_USGS_RETRIES`` said, silently capping the feature this + matter what ``API_USGS_RETRIES`` said, capping without any signal the feature this layer exists to provide. """ attempts = 0 @@ -396,7 +397,7 @@ def sleep(seconds: float) -> None: retry.RetryPolicy(max_retries=4, stall_timeout=60.0, retry_after_cap=60.0), ) - assert attempts == 5, "a sanctioned wait costs the no-progress budget nothing" + assert attempts == 5, "a server-named wait costs the no-progress budget nothing" def test_credited_wait_never_credits_past_the_present(monkeypatch) -> None: @@ -404,24 +405,24 @@ def test_credited_wait_never_credits_past_the_present(monkeypatch) -> None: ``credit_wait`` moves the progress stamp forward; without a ceiling at "now", one long queue wait pushed it into the future, made - ``elapsed_since_progress`` negative, and -- since nothing ever pulls it back - -- left that call exempt from the stall bound for the rest of its life. + ``elapsed_since_progress`` negative, and -- since nothing ever reduces it + -- left that call exempt from the stall bound until it returned. """ now = 0.0 monkeypatch.setattr(liveness.time, "monotonic", lambda: now) policy = retry.RetryPolicy(stall_timeout=60.0) liveness.note_progress() - liveness.credit_wait(300.0) # a deep-tail task queued past the whole budget + liveness.credit_wait(300.0) # a task queued past the whole budget assert liveness.elapsed_since_progress() == 0.0, "clamped to now, not negative" - # The budget is spent again by real silence, not permanently disabled. + # The budget is spent again by real time without data, not permanently disabled. now = 200.0 assert not policy.allows_wait(5, 30.0, liveness.elapsed_since_progress()) def test_arriving_pages_restart_the_stall_budget(monkeypatch) -> None: - """A slow but productive download keeps earning more time.""" + """A slow but productive download keeps restarting the budget.""" now = 0.0 monkeypatch.setattr(liveness.time, "monotonic", lambda: now) policy = retry.RetryPolicy(stall_timeout=60.0) @@ -435,7 +436,7 @@ def test_arriving_pages_restart_the_stall_budget(monkeypatch) -> None: def test_bad_retry_environment_raises_a_catchable_error(monkeypatch) -> None: - """A typo in the environment must not escape as a bare ValueError. + """A typo in the environment must not propagate as a bare ValueError. Every retrieval path builds its policy from the environment, so an unparseable value would otherwise bypass ``except DataRetrievalError`` in @@ -458,11 +459,11 @@ def test_bad_retry_environment_raises_a_catchable_error(monkeypatch) -> None: def test_queued_work_keeps_its_retries() -> None: - """Time spent waiting for a concurrency slot is not silence. + """Time spent waiting for a concurrency slot is not time without data. The no-progress budget starts when a retry loop is entered, but a fan-out - task may sit behind a full semaphore long after that. Without excusing the - wait, the tail of a wide fan-out enters its first attempt with the budget + task may wait on a full semaphore long after that. Without crediting the + wait, the last chunks of a wide fan-out enter their first attempt with the budget already spent, while the tasks dispatched ahead of it get the full allowance. """ @@ -500,16 +501,16 @@ async def attempt() -> str: def test_gate_does_not_reset_silence_from_earlier_attempts() -> None: - """Excusing the queue wait must not also forgive accumulated silence. + """Crediting the queue wait must not also discard accumulated time without data. The gated body is what the retry loop re-invokes, so stamping "now" on every - slot acquisition would restart the clock each attempt and quietly turn a - bound on *total* silence into a per-attempt latency bound -- five slow - failures would each look brief while the call sat silent for their sum. + slot acquisition would restart the budget each attempt and turn a bound on + *total* time without data into a per-attempt latency bound -- five slow + failures would each look brief while the call received nothing for their sum. """ async def drive() -> int: - gate = asyncio.Semaphore(4) # never contended: no waiting to excuse + gate = asyncio.Semaphore(4) # never contended: no waiting to credit attempts = 0 policy = retry.RetryPolicy( max_retries=4, stall_timeout=1.0, base_backoff=0.001, max_backoff=0.001 @@ -518,7 +519,7 @@ async def drive() -> int: async def attempt() -> str: nonlocal attempts attempts += 1 - await asyncio.sleep(0.4) # each attempt is silent for 0.4 s + await asyncio.sleep(0.4) # each attempt receives nothing for 0.4 s raise ServiceUnavailable("gateway", status_code=504) try: @@ -553,11 +554,11 @@ def _wrapped_dns_failure(errno: int) -> NetworkError: def test_deterministic_failures_are_not_offered_as_resumable() -> None: """ "Retryable" and "resumable" are one judgement and must agree. - ``_retryable`` already refuses to re-send a hostname the resolver rejects + ``_retryable`` already does not re-send a hostname the resolver rejects outright. If the interruption classifier still mapped it to - ``ServiceInterrupted``, the caller would be handed a ``.call.resume()`` + ``ServiceInterrupted``, the caller would be given a ``.call.resume()`` whose every attempt fails identically -- a resumable handle for something - that cannot be resumed, hiding the ``NetworkError`` that actually explains + that cannot be resumed, concealing the ``NetworkError`` that explains the failure. """ from dataretrieval.interruptions import _classify_chunk_error @@ -570,9 +571,9 @@ def test_deterministic_failures_are_not_offered_as_resumable() -> None: assert retry._retryable(unsupported) == (False, None) assert _classify_chunk_error(unsupported) is None - # The converse still holds: a failure a later attempt could survive stays - # both retryable and resumable. A temporary resolver failure is the sharp - # case -- same exception type, same chain shape, opposite verdict, decided + # The converse still holds: a failure a later attempt could succeed on stays + # both retryable and resumable. A temporary resolver failure is the distinguishing + # case -- same exception type, same chain shape, opposite classification, decided # only by the errno. temporary = _wrapped_dns_failure(socket.EAI_AGAIN) assert retry._retryable(temporary) == (True, None) @@ -580,13 +581,13 @@ def test_deterministic_failures_are_not_offered_as_resumable() -> None: def test_exception_chain_walk_terminates_on_a_self_referencing_chain() -> None: - """Every question asked of a failure chain shares one guarded traversal. + """Every classification of a failure chain shares one guarded traversal. ``raise ... from`` accepts an exception already in the chain, so a retry loop that re-raises an earlier failure can close the cycle. Each of these walks reaches an answer by inspecting links, so an unguarded one would spin - forever inside a request path rather than surface the failure. This fails by - hanging, not by asserting -- pytest's timeout is the real assertion. + forever inside a request path rather than raise the failure. This fails by + hanging, not by asserting -- pytest's timeout is what detects the failure. """ from dataretrieval.interruptions import ( ServiceInterrupted, @@ -603,7 +604,7 @@ def test_exception_chain_walk_terminates_on_a_self_referencing_chain() -> None: assert {id(exc) for exc in _walk_causes(first)} == {id(first), id(second)} assert _classify_chunk_error(first) is None assert _deterministic_failure(first) is False - # The status hunt in the interruption constructor walks the same chain. + # The status lookup in the interruption constructor walks the same chain. assert ( ServiceInterrupted(completed_chunks=0, total_chunks=1, cause=first).status_code is None @@ -612,7 +613,7 @@ def test_exception_chain_walk_terminates_on_a_self_referencing_chain() -> None: def test_an_unusable_next_page_link_is_reported_not_swallowed(): """A malformed ``next`` href ends the page walk, so it must raise rather - than quietly truncate the result -- a short frame with no error is + than truncate the result without an error -- a short frame with no error is indistinguishable from a complete one.""" from dataretrieval.transport.links import resolve_next_url @@ -628,15 +629,15 @@ def test_an_unusable_next_page_link_is_reported_not_swallowed(): class TestErrorForStatus: def test_a_success_status_is_a_usage_error(self): - """``error_for_status`` builds an exception for a failure. Handing it a - 200 means the caller took the error branch on a success status, and - returning some default exception would hide that.""" + """``error_for_status`` builds an exception for a failure. Passing it a 200 + means the caller took the error branch on a success status, and + returning some default exception would conceal that.""" with pytest.raises(ValueError, match="expects an HTTP error status"): exceptions.error_for_status(200, "not an error") def test_a_leaf_without_a_default_status_demands_one(self): - """Only RateLimited and ServiceUnavailable imply their own status. Any - other HTTPError constructed without one would carry a meaningless - status_code, so it refuses instead.""" + """Only RateLimited and ServiceUnavailable imply their own status. Any other + HTTPError constructed without one would have a meaningless status_code, so + construction raises instead.""" with pytest.raises(TypeError, match="requires status_code"): exceptions.TransientError("boom") diff --git a/tests/utils_test.py b/tests/utils_test.py index fa5521d43..c2fcb6c26 100644 --- a/tests/utils_test.py +++ b/tests/utils_test.py @@ -32,7 +32,7 @@ class Test_query: def test_url_too_long(self, httpx_mock): """A 413 / 414 from the service (an over-long query URL, Issue #64) is - surfaced as the typed URLTooLong.""" + raised as the typed URLTooLong.""" httpx_mock.add_response(method="GET", status_code=414) with pytest.raises(exceptions.URLTooLong): nwis.get_iv(sites=["01491000", "01491001"]) @@ -67,7 +67,7 @@ def test_no_sites_detection_respects_response_charset(self, httpx_mock): utils.query(url, {}) def test_query_does_not_opt_into_retry(self, httpx_mock, monkeypatch): - """The public legacy adapter still surfaces the first transient failure.""" + """The public legacy adapter still raises the first transient failure.""" url = "https://example.invalid/x" request_url = f"{url}?a=1" httpx_mock.add_response(method="GET", url=request_url, status_code=503) @@ -101,7 +101,7 @@ class Test_error_taxonomy: """The unified request-error hierarchy. Every module's request failure is catchable as ``DataRetrievalError``. - A status error is an ``HTTPError`` carrying ``.status_code`` (the retryable + A status error is an ``HTTPError`` with ``.status_code`` (the retryable 429 / 5xx subset is ``TransientError``); a connection failure is a ``NetworkError``. The sole base is ``DataRetrievalError`` -- no builtin (``ValueError`` / ``RuntimeError``) mixins. @@ -119,7 +119,7 @@ class Test_error_taxonomy: ) def test_query_maps_status_to_typed_error(self, httpx_mock, status, exc_name): """``query`` maps each HTTP status to the right typed ``DataRetrievalError``: - a generic ``HTTPError`` (carrying ``.status_code``) for a fatal 4xx, and + a generic ``HTTPError`` (with ``.status_code``) for a fatal 4xx, and the transient ``RateLimited`` / ``ServiceUnavailable`` for 429 / 5xx. The too-long-URL statuses (413 / 414) are covered separately because their message is the actionable remediation, not the bare status number.""" @@ -134,7 +134,7 @@ def test_query_maps_status_to_typed_error(self, httpx_mock, status, exc_name): @pytest.mark.parametrize("status", [413, 414]) def test_query_too_long_url_gives_actionable_message(self, httpx_mock, status): - """A server 413 / 414 surfaces as ``URLTooLong`` carrying the actionable + """A server 413 / 414 is raised as ``URLTooLong`` with the actionable "Modify your query" remediation (the same message as the client-side over-long-URL path), not a bare ``HTTP 414`` status line.""" url = "https://example.invalid/x" @@ -144,7 +144,7 @@ def test_query_too_long_url_gives_actionable_message(self, httpx_mock, status): assert isinstance(excinfo.value, exceptions.RequestTooLarge) def test_transport_error_wrapped_as_network_error(self, httpx_mock): - """A connection-level failure (no HTTP response) surfaces as the typed + """A connection-level failure (no HTTP response) is raised as the typed ``NetworkError`` -- catchable via ``except DataRetrievalError`` like the response-based errors, with the original ``httpx`` exception on ``__cause__`` -- rather than leaking a raw ``httpx`` exception.""" @@ -188,7 +188,7 @@ def test_uniform_retry_attributes_readable_on_every_error(self): assert err.retryable is retryable, err def test_no_sites_error_is_data_retrieval_error(self): - """``NoSitesError`` (the legacy nwis no-data signal) roots at + """``NoSitesError`` (the legacy nwis no-data signal) derives from ``DataRetrievalError`` and is not a builtin ``ValueError``, so it is caught by the unified ``except dataretrieval.DataRetrievalError``.""" assert issubclass(exceptions.NoSitesError, exceptions.DataRetrievalError) @@ -200,7 +200,7 @@ def test_no_sites_error_is_data_retrieval_error(self): def test_typed_errors_survive_pickle_and_deepcopy(self): """Typed errors round-trip through pickle/deepcopy -- they get pickled back from multiprocessing / lithops workers, and their constructor fields - (status_code, retry_after, url) must survive the trip.""" + (status_code, retry_after, url) must be preserved.""" import copy import pickle @@ -228,7 +228,7 @@ def test_typed_errors_survive_pickle_and_deepcopy(self): def test_waterdata_exceptions_share_the_root(self): """waterdata's typed exceptions are ``DataRetrievalError`` too, so one ``except`` clause spans the legacy and waterdata subsystems, and they - slot under the shared family bases (``HTTPError`` / ``TransientError`` / + are subclasses of the shared base classes (``HTTPError`` / ``TransientError`` / ``RequestTooLarge``).""" from dataretrieval.exceptions import ( RateLimited, @@ -243,7 +243,7 @@ def test_waterdata_exceptions_share_the_root(self): assert issubclass(RateLimited, exceptions.TransientError) assert issubclass(ServiceUnavailable, exceptions.TransientError) assert issubclass(ServiceUnavailable, exceptions.HTTPError) - # "Too large" failures slot under RequestTooLarge. + # "Too large" failures are subclasses of RequestTooLarge. assert issubclass(Unchunkable, exceptions.RequestTooLarge) def test_base_exported_at_top_level(self): @@ -521,8 +521,9 @@ def test_the_table_names_no_endpoint_parameter_of_its_own(self): class TestTerritories: - """The five territories are real ANSI/FIPS entities, and every service this - package reaches carries data for them.""" + """The five territories are real ANSI/FIPS entities, and every service this package + queries has data for them. + """ @pytest.mark.parametrize( ("value", "name", "postal", "fips"), @@ -579,9 +580,9 @@ def test_the_remedy_names_this_endpoints_native_parameters(self): assert "using the API's native value" in message def test_an_endpoint_with_no_alternative_offers_none(self): - """NGWMN's getters expose only the unified ``state``, so appending a - remedy from ``into`` sent a caller to ``get_sites(state_name=...)`` - (``TypeError``) or straight back into this same error.""" + """NGWMN's getters expose only the unified ``state``, so appending a remedy from + ``into`` directed a caller to ``get_sites(state_name=...)`` (``TypeError``) or + back to this same error.""" from dataretrieval.codes.states import apply_state for into, to in (("state", "postal"), ("state_name", "name")): @@ -628,7 +629,7 @@ def test_joins_date_time_and_zone_into_utc(self): def test_warns_and_keeps_going_when_a_timestamp_will_not_parse(self): """An unparseable row becomes NaT rather than failing the whole frame, - but silently dropping timestamps would return an incomplete frame with + but dropping timestamps would return an incomplete frame with nothing to mark the gap -- so it warns, and names the switch that avoids the loss.""" df = pd.DataFrame( @@ -650,8 +651,8 @@ def test_warns_and_keeps_going_when_a_timestamp_will_not_parse(self): def test_base_metadata_repr_names_the_type_and_url(): - """``md`` is what a user prints when a query surprises them, so the repr - has to say which metadata class it is and which URL produced it.""" + """``md`` is what a user prints when a query returns something unexpected, so + the repr has to say which metadata class it is and which URL produced it.""" response = mock.MagicMock() response.url = "https://example.test/items?limit=1" md = utils.BaseMetadata(response) diff --git a/tests/validation_test.py b/tests/validation_test.py index 96d2c5f9c..22d5a4ccd 100644 --- a/tests/validation_test.py +++ b/tests/validation_test.py @@ -1,7 +1,7 @@ """Tests for the shared argument checks. Each check is asserted on two things: that it lets a valid call through, and -that its rejection names the move that would fix the call. The second half is +that its rejection names the change that would fix the call. The second half is the point of the module -- a caller that is a program can only correct itself from a message that says what to send instead. """ @@ -37,7 +37,7 @@ def test_context_qualifies_a_vocabulary_that_depends_on_another_argument(): def test_remedy_adds_a_move_without_dropping_the_options(): """A vocabulary narrower than the service's needs both halves: what this - function takes, and how to reach the rest.""" + function takes, and how to obtain the rest.""" with pytest.raises(ValueError) as excinfo: require_one_of( "hourly", ("daily",), name="collection", remedy="Call get_queryables." @@ -49,8 +49,8 @@ def test_remedy_adds_a_move_without_dropping_the_options(): def test_a_string_vocabulary_is_refused(): """``str`` is a Collection, so passing one type-checks -- and then ``in`` - silently degrades from membership to a substring test, accepting any - fragment of a valid option. Refuse it at the one shared chokepoint.""" + changes from a membership test to a substring test without any error, accepting any + fragment of a valid option. Refuse it in the one shared check.""" with pytest.raises(TypeError, match="not 'csv'"): require_one_of("cs", "csv", name="format") @@ -84,7 +84,7 @@ def test_accepts_all_supplied(self): require_together({"lat": 1.0, "long": 2.0}) def test_accepts_none_supplied(self): - """Declining the whole group is a different question from completing it.""" + """Omitting the whole group is a different case from completing it.""" require_together({"lat": None, "long": None}) def test_message_names_what_is_missing_and_what_to_do(self): @@ -157,7 +157,7 @@ def test_accepts_one_supplied(self): reject_together({"lat": 1.0, "comid": None}) def test_accepts_none_supplied(self): - """Unlike require_exactly_one, an empty call is not this check's business.""" + """Unlike require_exactly_one, an empty call is not this check's concern.""" reject_together({"lat": None, "comid": None}) def test_message_names_only_the_conflicting_arguments(self): diff --git a/tests/waterdata_chunking_test.py b/tests/waterdata_chunking_test.py index 092641c77..810456ffa 100644 --- a/tests/waterdata_chunking_test.py +++ b/tests/waterdata_chunking_test.py @@ -11,7 +11,7 @@ The one exception is ``test_joint_planner_url_construction_long_filter_and_long_sites``, which uses the real ``_construct_api_requests`` so URL-encoding -surprises (``%``, ``+``, ``/``, ``&``, …) can't pass against a fake +differences (``%``, ``+``, ``/``, ``&``, …) can't pass against a fake and then fail in production. """ @@ -86,7 +86,7 @@ from dataretrieval.utils import HTTPX_DEFAULTS from dataretrieval.waterdata.utils import OGC_API_URL -# The joint-planner stress test drives the real request builder; bind the +# The joint-planner stress test uses the real request builder; bind the # target API the way ``get_ogc_data`` does — explicitly, via ``partial``. _construct_api_requests = functools.partial( _construct_api_requests_explicit, base_url=OGC_API_URL @@ -132,7 +132,7 @@ def __init__(self, url, content=b""): def _fake_build(*, base=200, **kwargs): """Fake build_request: URL length deterministic in its inputs. - Mirrors the GET-routed shape: payload goes in the URL, body is empty. + Matches the GET-routed shape: payload goes in the URL, body is empty. List/string values are URL-encoded via ``quote_plus`` so the fake's byte count matches what the real ``_construct_api_requests`` would produce; otherwise an alphanumeric test could pass against the fake @@ -154,7 +154,7 @@ def test_never_chunk_covers_all_date_range_params(): but every date-range param MUST be excluded from chunking — a range value isn't an enumerable set to split. Guard against drift: adding a new param to ``_DATE_RANGE_PARAMS`` without also adding - it to ``_NEVER_CHUNK`` would silently let the chunker try to + it to ``_NEVER_CHUNK`` would let the chunker try to comma-join an interval string.""" missing = _DATE_RANGE_PARAMS - _NEVER_CHUNK assert not missing, ( @@ -206,8 +206,8 @@ def test_chunk_plan_returns_passthrough_when_no_chunkable_axes(): def test_chunk_plan_raises_when_unchunkable_request_exceeds_limit(): """A request with nothing to chunk that still exceeds the byte limit (e.g. a single large CQL ``IN`` clause with no top-level ``OR``) raises - Unchunkable instead of being shipped for the server to reject with an - opaque HTTP 414.""" + Unchunkable instead of being sent for the server to reject with an + uninformative HTTP 414.""" args = {"monitoring_location_id": "scalar-only"} with pytest.raises(Unchunkable): ChunkPlan(args, _fake_build, url_limit=10) @@ -216,8 +216,9 @@ def test_chunk_plan_raises_when_unchunkable_request_exceeds_limit(): def test_chunk_plan_passes_through_unchunkable_cql_json_over_limit(): """A cql-json filter is outside the chunker's domain (it splits only cql-text), so an over-budget cql-json request is passed through unchanged - instead of raising — the server judges it, not us. Guards against the - chunker hijacking the deliberate cql-json passthrough.""" + instead of raising — the server accepts or rejects it, not this package. + Guards against the chunker overriding the deliberate cql-json passthrough. + """ args = {"filter": "a OR b OR c", "filter_lang": "cql-json"} plan = ChunkPlan(args, _fake_build, url_limit=10) assert plan.axes == [] @@ -232,7 +233,7 @@ def test_chunk_plan_greedy_halving_targets_largest_axis_chunk(): "monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30], "parameter_code": ["00060", "00065"], } - # full URL ≈ 200 + 123 + 12 = 335; force splitting the heavy axis only. + # full URL ≈ 200 + 123 + 12 = 335; force splitting the largest axis only. plan = ChunkPlan(args, _fake_build, url_limit=310) assert len(plan.chunks["monitoring_location_id"]) > 1 assert len(plan.chunks["parameter_code"]) == 1 @@ -274,7 +275,7 @@ def test_chunk_plan_minimizes_total_chunks(): "sites": ["S" * 30 for _ in range(8)], # 8 sites @ 30 chars "filter": " OR ".join(clauses), } - # Tight limit forces both axes to participate. + # A small limit forces both axes to participate. plan = ChunkPlan(args, _fake_build, url_limit=380) # Plan must stay under the all-singleton worst case (8 singletons × 16 # filter chunks = 128 chunks). @@ -282,8 +283,8 @@ def test_chunk_plan_minimizes_total_chunks(): def test_chunk_plan_raises_when_smallest_plan_doesnt_fit(): - """If even the most aggressive joint plan (singleton lists + - singleton filter clauses) still exceeds the limit, surface + """If even the smallest joint plan (singleton lists + + singleton filter clauses) still exceeds the limit, raise Unchunkable — there's nothing left to shrink.""" args = { "monitoring_location_id": ["A" * 10, "B" * 10], @@ -391,14 +392,14 @@ async def fetch(args): monkeypatch.setattr(_chunking, "_OGC_URL_BYTE_LIMIT", 240) # 4 sites of 10 chars → exceeds 240 → planner splits. fetch({"sites": ["S" * 10 + str(i) for i in range(4)]}) - assert len(calls) > 1, "patched constant should drive chunking" + assert len(calls) > 1, "patched constant should control chunking" def test_chunked_session_shared_across_chunks(): - """Every chunk of one chunked call sees the same - ``httpx.AsyncClient`` on the ``_chunked_client`` ContextVar, so - downstream paginated helpers (``_walk_pages``) can reuse the - connection pool instead of handshaking fresh on each chunk.""" + """Every chunk of one chunked call uses the same ``httpx.AsyncClient`` on the + ``_chunked_client`` ContextVar, so downstream paginated helpers (``_walk_pages``) + can reuse the + connection pool instead of opening a new connection on each chunk.""" sessions_seen = [] @multi_value_chunked(build_request=_fake_build, url_limit=240) @@ -408,15 +409,15 @@ async def fetch(args): elapsed=datetime.timedelta(seconds=0.1), headers={} ) - # Outside a chunked call: no session published (in this thread/context). + # Outside a chunked call: no session set (in this thread/context). assert _chunked_client.get() is None fetch({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) - # Plan must actually fan out — otherwise the test isn't exercising + # Plan must fan out — otherwise the test isn't exercising # the shared-session path. assert len(sessions_seen) > 1 - # Every chunk saw a Session, not None. + # Every chunk received a Session, not None. assert all(s is not None for s in sessions_seen) # And it was the same object every time. assert len({id(s) for s in sessions_seen}) == 1 @@ -426,10 +427,9 @@ async def fetch(args): def test_chunked_session_isolated_per_resume(): - """A follow-up ``resume`` after an interruption opens a fresh - session — the previous one was closed when its ``resume`` returned. - The ContextVar is reset between runs so leakage can't carry - a closed session into the retry.""" + """A follow-up ``resume`` after an interruption opens a new session — the previous + one was closed when its ``resume`` returned. The ContextVar is reset between runs so + a closed session cannot be carried into the retry.""" state = {"i": 0, "blow_up": True} sessions_seen = [] @@ -451,7 +451,7 @@ async def fetch(args): with pytest.raises(QuotaExhausted) as excinfo: fetch({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) - # First run published a shared client to its chunks; the calling + # First run set a shared client for its chunks; the calling # thread's ContextVar is unaffected (reads its default). assert _chunked_client.get() is None first_run_sessions = list(sessions_seen) @@ -461,7 +461,7 @@ async def fetch(args): excinfo.value.call.resume() # Second run's ContextVar is also reset in the calling thread. assert _chunked_client.get() is None - # The resume opened a FRESH client, distinct from the first run's, so no + # The resume opened a NEW client, distinct from the first run's, so no # closed client leaks across runs. resume_sessions = sessions_seen[len(first_run_sessions) :] assert resume_sessions and all(s is not None for s in resume_sessions) @@ -479,8 +479,8 @@ def _quota_response(remaining: int | str | None) -> mock.Mock: def test_quota_exhausted_on_mid_call_429(): - """Mid-call 429 (a concurrent caller drained the window) surfaces - as ``QuotaExhausted`` carrying the partial frame plus the chunk + """Mid-call 429 (a concurrent caller exhausted the quota window) is raised + as ``QuotaExhausted`` with the partial frame plus the chunk offset so callers can resume after the window resets.""" state = {"i": 0} @@ -516,7 +516,7 @@ async def fetch(args): def test_quota_exhausted_on_first_chunk_429_has_no_partial_response(): - """A 429 on the very first chunk means no responses have + """A 429 on the first chunk means no responses have completed; ``partial_response`` is ``None`` (and ``partial_frame`` is empty) so callers can branch on that to distinguish "abort before any data arrived" from "abort after partial collection".""" @@ -562,7 +562,7 @@ async def fetch(args): decorated = multi_value_chunked(build_request=_fake_build, url_limit=240)(fetch) sites = ["S1" * 10, "S2" * 10, failing_site, "S4" * 10, "S5" * 10] - # First attempt: 429 on the chunk carrying the failing site; the other + # First attempt: 429 on the chunk containing the failing site; the other # four chunks complete. with pytest.raises(QuotaExhausted) as excinfo: decorated({"sites": sites}) @@ -585,10 +585,10 @@ async def fetch(args): def test_quota_exhausted_resume_can_reraise_on_persistent_429(): - """If the window is still empty when the caller resumes, + """If the quota window has not reset when the caller resumes, ``call.resume()`` raises ``QuotaExhausted`` again — the - ``ChunkedCall``'s in-flight state carries forward, so a - subsequent resume after a longer wait still picks up from the pending chunk.""" + ``ChunkedCall``'s in-flight state is kept, so a + subsequent resume after a longer wait still resumes from the pending chunk.""" # Key the failure on the chunk's CONTENT (one persistently-429ing # site) rather than a global call counter: under the async fan-out # every other chunk completes, and the same still-pending @@ -620,7 +620,7 @@ async def fetch(args): def test_resume_produces_dataset_identical_to_uninterrupted_run(): """End-to-end resume equivalence: the same chunked query run twice - — once straight through, once with a mid-stream 429 + + — once straight through, once with a 429 partway through + ``call.resume()`` — must yield byte-identical combined frames. Guards against off-by-one errors in the resume cursor (re-fetching the chunk that 429'd, or skipping past it) and any ordering drift @@ -667,13 +667,13 @@ async def fetch(args): decorated_b = multi_value_chunked(build_request=_fake_build, url_limit=240)(fetch_b) with pytest.raises(QuotaExhausted) as excinfo: decorated_b({"sites": sites}) - # The 429 must hit mid-stream — otherwise the test isn't exercising - # what we think it is. + # The 429 must occur partway through — otherwise the test is not exercising the + # resume path. assert 0 < excinfo.value.completed_chunks < excinfo.value.total_chunks df_b, _ = excinfo.value.call.resume() - # Sanity: both runs must have actually chunked (otherwise the - # 429-mid-stream branch wasn't exercised). + # Check: both runs must have chunked (otherwise the + # 429-partway branch wasn't exercised). assert excinfo.value.total_chunks > 1 # The combined DataFrames must be byte-identical: same rows in the @@ -690,12 +690,12 @@ def test_resume_rebuilds_chunks_from_creation_time_bindings(): """Regression: everything a chunk rebuild needs (base URL, dialect, row cap in production) is bound into the ``fetch``/``build_request`` closures when the call is created — ``get_ogc_data`` binds them with - ``functools.partial``. A ``call.resume()`` fired AFTER the originating - call returned — the documented recovery for a mid-stream 429 — re-issues + ``functools.partial``. A ``call.resume()`` invoked AFTER the originating + call returned — the documented recovery for a 429 partway through — re-issues the pending chunks through those same bound callables, so every rebuilt chunk observes the creation-time values. A reintroduced ambient read in the fetch path would break this without any executor-side snapshot to - paper over it.""" + mask it.""" observed: list[str] = [] state = {"calls": 0, "tripped": False} @@ -723,8 +723,7 @@ async def fetch(args, *, base): assert 0 < excinfo.value.completed_chunks < excinfo.value.total_chunks # Resume outside the originating call. Every rebuilt chunk must be - # fetched through the closure created at call time, still carrying the - # bound base. + # fetched through the closure created at call time, still holding the bound base. observed.clear() df, _ = excinfo.value.call.resume() assert observed, "resume issued no chunks" @@ -733,13 +732,13 @@ async def fetch(args, *, base): def test_resume_reads_concurrency_from_the_caller_not_the_snapshot(monkeypatch): - """A ``configure()`` block around a ``resume()`` must actually take effect. + """A ``configure()`` block around a ``resume()`` must take effect. - The concurrency cap is the one dial a caller adjusts precisely *when* + The concurrency cap is the one setting a caller adjusts *when* retrying -- the documented recovery from ``QuotaExhausted`` is to wait and - re-issue more gently -- so ``resume()`` resolves it per drive rather than - carrying a value fixed when the call was constructed. A ``configure()`` - block entered between the interruption and the resume therefore wins. + re-issue at lower concurrency -- so ``resume()`` resolves it per run rather than + holding a value fixed when the call was constructed. A ``configure()`` + block entered between the interruption and the resume therefore takes precedence. """ state = {"calls": 0} @@ -773,7 +772,7 @@ async def spy_run(self, max_concurrent): def test_chunker_passes_through_non_429_runtime_error(): """A non-429 ``RuntimeError`` (e.g. a 500) is not a quota signal; - it must propagate unchanged so callers see the real cause.""" + it must propagate unchanged so callers get the real cause.""" state = {"i": 0} async def fetch(args): @@ -794,7 +793,7 @@ async def fetch(args): def test_chunker_wraps_service_unavailable_as_resumable(): """A typed ``ServiceUnavailable`` (HTTP 5xx) is a transient transport failure: ``ChunkedCall`` must wrap it as - ``ServiceInterrupted`` carrying the partial state, parallel to how + ``ServiceInterrupted`` with the partial state, parallel to how a 429 becomes ``QuotaExhausted``. Once the upstream recovers, ``.call.resume()`` resumes only the still-pending chunks.""" state = {"i": 0, "blow_up": True} @@ -819,7 +818,7 @@ async def fetch(args): # Resumable: handle on .call with already-completed work preserved. assert err.call is not None # Async fan-out: only the i==2 chunk fails; the gather completes - # the other four, so 4 of 5 are recorded before the failure surfaces. + # the other four, so 4 of 5 are recorded before the failure is raised. assert err.completed_chunks == 4 assert err.total_chunks == 5 assert not err.call.partial_frame.empty @@ -836,7 +835,7 @@ def test_chunk_interrupted_base_class_catches_both(): and ``ServiceInterrupted`` must both subclass it.""" assert issubclass(QuotaExhausted, ChunkInterrupted) assert issubclass(ServiceInterrupted, ChunkInterrupted) - # ``ChunkInterrupted`` roots at ``DataRetrievalError`` like the rest of the + # ``ChunkInterrupted`` derives from ``DataRetrievalError`` like the rest of the # taxonomy (no ``RuntimeError`` mixin), so one ``except DataRetrievalError`` # spans chunked and single-shot failures alike. assert issubclass(ChunkInterrupted, DataRetrievalError) @@ -844,22 +843,22 @@ def test_chunk_interrupted_base_class_catches_both(): def test_chunk_interrupted_pickles_as_degraded_across_process_boundary(): - """A real ChunkInterrupted carries a live ChunkedCall whose ``fetch`` is not + """A real ChunkInterrupted holds a live ChunkedCall whose ``fetch`` is not stdlib-picklable, so a worker raising it inside a multiprocessing / - ProcessPoolExecutor pool could not ship it back. ``__getstate__`` drops + ProcessPoolExecutor pool could not return it. ``__getstate__`` drops ``.call`` and pickles the documented degraded ``call=None`` state -- counts - and retry hint preserved, ``.resume()`` gone (un-resumable cross-process).""" + and retry-after value preserved, ``.resume()`` gone (un-resumable cross-process).""" import pickle plan = ChunkPlan( {"monitoring_location_id": ["A", "B", "C"]}, _fake_build, url_limit=8000 ) - # A local function isn't picklable by reference -- mirrors production, where + # A local function isn't picklable by reference -- matches production, where # ChunkedCall.fetch is the undecorated _fetch_once shadowed by its wrapper. call = ChunkedCall(plan, lambda args: (pd.DataFrame(), None)) exc = call.wrap_failure(RateLimited("429: too many requests", retry_after=12.0)) assert isinstance(exc, QuotaExhausted) and exc.call is call - # the live fetch handle alone can't pickle (the whole point of the override) + # the live fetch handle alone can't pickle (what the override exists for) with pytest.raises((pickle.PicklingError, AttributeError)): pickle.dumps(exc.call.fetch) @@ -874,7 +873,7 @@ def test_chunk_interrupted_pickles_as_degraded_across_process_boundary(): def test_chunk_interrupted_with_partial_data_pickles_intact(): """The degrade drops only the live ``.call``; the captured *partial work* - must still cross the boundary so a worker can report what it salvaged. + must still cross the boundary so a worker can report what it completed. Exercises the path the no-completed-chunks case above doesn't: a real ``partial_frame`` (rows) and ``partial_response`` (a live ``httpx.Response``, which must itself remain picklable).""" @@ -907,7 +906,7 @@ def test_chunk_interrupted_with_partial_data_pickles_intact(): def test_connection_error_wrapped_as_service_interrupted(): """A bare ``httpx.ConnectError`` (or any other transport-level ``httpx.HTTPError``) doesn't inherit from ``RuntimeError``; - without the widened catch in ``_issue`` it would escape uncaught + without the widened catch in ``_issue`` it would propagate uncaught and the user would lose the resumable handle to ``.call.resume()``. Verify ``ChunkedCall`` wraps it as ``ServiceInterrupted`` so partial progress is preserved.""" @@ -931,7 +930,7 @@ async def fetch(args): # Async fan-out: only the i==2 chunk fails; the other four complete. assert err.completed_chunks == 4 assert err.call is not None - # The transport exception is on __cause__ so callers can drill in if needed. + # The transport exception is on __cause__ so callers can inspect it if needed. assert isinstance(err.__cause__, httpx.ConnectError) # Resume after the upstream recovers. state["blow_up"] = False @@ -942,9 +941,9 @@ async def fetch(args): def test_invalid_url_wrapped_as_service_interrupted(): """``httpx.InvalidURL`` inherits from ``Exception``, NOT from ``httpx.HTTPError``. Without the widened catch in ``_issue`` / - ``_classify_chunk_error`` an oversize follow-up URL escapes as + ``_classify_chunk_error`` an oversize follow-up URL propagates as raw ``InvalidURL`` and the user loses ``.call.resume()`` access - to the partial state. Mirror the ConnectError test.""" + to the partial state. Match the ConnectError test.""" state = {"i": 0, "blow_up": True} async def fetch(args): @@ -966,9 +965,9 @@ async def fetch(args): assert err.completed_chunks == 4 assert err.call is not None assert isinstance(err.__cause__, httpx.InvalidURL) - # The top-level message must surface the underlying cause text so + # The top-level message must include the underlying cause text so # the user doesn't have to traverse ``__cause__`` to know what - # actually failed (previously the message was generic "Service + # failed (previously the message was generic "Service # error after K/N chunks; ... resume() once the upstream # recovers", with the real "URL too long" only visible via # ``.__cause__``). @@ -977,11 +976,10 @@ async def fetch(args): def test_service_interrupted_exposes_partial_frame_and_response(): - """Both ``QuotaExhausted`` AND ``ServiceInterrupted`` carry - ``partial_frame`` / ``partial_response`` directly on the - exception. Previously only ``QuotaExhausted`` had them, so a - generic ``except ChunkInterrupted as exc: log(exc.partial_frame)`` - crashed with AttributeError on 5xx.""" + """Both ``QuotaExhausted`` AND ``ServiceInterrupted`` have ``partial_frame`` / + ``partial_response`` directly on the exception. Previously only ``QuotaExhausted`` + had them, so a generic ``except ChunkInterrupted as exc: log(exc.partial_frame)`` + raised AttributeError on 5xx.""" state = {"i": 0} async def fetch(args): @@ -1009,12 +1007,11 @@ async def fetch(args): def test_partial_frame_snapshot_stable_across_resume(): - """``exc.partial_frame`` / ``exc.partial_response`` snapshot the - state at raise time. Calling ``exc.call.resume()`` advances the - underlying ``ChunkedCall`` but must NOT mutate the snapshot on - the exception — otherwise a diagnostic that reads - ``exc.partial_frame`` after a resume sees post-resume state under - a name that promises pre-resume state.""" + """``exc.partial_frame`` / ``exc.partial_response`` snapshot the state at raise + time. Calling ``exc.call.resume()`` advances the underlying ``ChunkedCall`` but must + NOT mutate the snapshot on the exception — otherwise a diagnostic that reads + ``exc.partial_frame`` after a resume sees post-resume state under a name that + denotes pre-resume state.""" state = {"i": 0, "blow_up": True} async def fetch(args): @@ -1045,13 +1042,13 @@ async def fetch(args): def test_partial_frame_snapshot_is_a_copy_when_single_chunk(): """``_combine_chunk_frames`` returns ``non_empty[0]`` verbatim on - its single-frame fast path. ``ChunkInterrupted.__init__`` must - therefore defensively ``.copy()`` so an in-place mutation of the - underlying chunk frame (e.g. user diagnostic code adding a - column on the live view) doesn't leak through the snapshot. - Companion to ``test_partial_frame_snapshot_stable_across_resume``, - which uses ≥2 completed chunks and so goes through - ``pd.concat`` (which already produces a fresh frame).""" + its single-frame fast path. ``ChunkInterrupted.__init__`` must therefore + ``.copy()`` so an in-place mutation of the underlying chunk frame (e.g. user + diagnostic code adding a column on the live view) doesn't leak through the + snapshot. Companion to ``test_partial_frame_snapshot_stable_across_resume``, + which uses ≥2 completed chunks and so goes through ``pd.concat`` (which + already produces a new frame). + """ state = {"i": 0, "blow_up": True} async def fetch(args): @@ -1064,7 +1061,7 @@ async def fetch(args): _quota_response(500), ) - # 2 sites at url_limit=240 → 2 singleton chunks. The 429 fires + # 2 sites at url_limit=240 → 2 singleton chunks. The 429 occurs # on the SECOND chunk and the gather completes the other, so the # exception captures exactly ONE completed chunk — the path where # _combine_chunk_frames aliases its single non-empty frame. @@ -1083,10 +1080,9 @@ async def fetch(args): def test_combine_chunk_responses_returns_independent_headers(): - """The aggregated response's ``.headers`` must be a fresh - ``httpx.Headers`` — mutations by downstream callers (logging - hooks, metadata extensions) must not back-propagate into the - underlying chunk response's headers, which still live on + """The aggregated response's ``.headers`` must be a new ``httpx.Headers`` — + mutations by downstream callers (logging hooks, metadata extensions) must not + back-propagate into the underlying chunk response's headers, which still live on ``ChunkedCall._chunks``.""" r0 = mock.Mock( elapsed=datetime.timedelta(seconds=0.1), headers={"X-Foo": "0"}, url="u0" @@ -1096,7 +1092,7 @@ def test_combine_chunk_responses_returns_independent_headers(): ) head = _combine_chunk_responses([r0, r1], canonical_url=None) - # Aggregate carries a chunk's headers (here the last, as the fallback when + # Aggregate has a chunk's headers (here the last, as the fallback when # neither reports a rate limit)... assert head.headers["X-Foo"] == "1" # ...but mutating the aggregate must not back-propagate. @@ -1106,8 +1102,8 @@ def test_combine_chunk_responses_returns_independent_headers(): def test_combine_chunk_responses_surfaces_lowest_remaining(): - """``x-ratelimit-remaining`` reports the LOWEST any chunk saw — the - quota actually left after the fan-out — not the last-by-index, which under + """``x-ratelimit-remaining`` reports the LOWEST any chunk received — the + quota left after the fan-out — not the last-by-index, which under concurrency need not be the response the server processed last.""" r0 = mock.Mock( elapsed=datetime.timedelta(seconds=0.1), @@ -1128,7 +1124,7 @@ def test_paginate_terminates_on_empty_string_cursor(): Parse-response wrappers in ``_walk_pages`` / ``stats.get_data`` coerce falsy non-None values to None so an empty-string next- cursor (a real-but-unusual end-of-stream sentinel some pagination - APIs use) doesn't trap us in an infinite ``follow_up('')`` loop.""" + APIs use) does not cause an infinite ``follow_up('')`` loop.""" # Synthesize an OGC response with numberReturned > 0 and a "next" # link whose href is an empty string — simulating a server-side # sentinel that ``_next_req_url`` reads as ``""``. @@ -1164,10 +1160,10 @@ def test_paginate_terminates_on_empty_string_cursor(): def test_combine_chunk_frames_does_not_collapse_none_ids(): """``drop_duplicates(subset='id')`` treats NaN==NaN as duplicate, so a blanket dedup would collapse every id-less row into one — - silent data loss. The function must dedupe only the id-bearing + undetected data loss. The function must dedupe only the id-bearing rows and preserve id-less rows verbatim.""" # Frame A has real ids; frame B has feature-IDs of None for two - # different rows that must both survive. + # different rows that must both be kept. df_a = pd.DataFrame({"id": ["x", "y"], "val": [1, 2]}) df_b = pd.DataFrame({"id": [np.nan, np.nan], "val": [3, 4]}) combined = _combine_chunk_frames([df_a, df_b]) @@ -1203,9 +1199,9 @@ async def fetch(args): def test_retry_after_surfaces_on_quota_exhausted(): """If the 429 response includes a ``Retry-After`` header, that - delay must travel from the typed transport exception + delay must be carried from the typed transport exception (``RateLimited.retry_after``) onto ``QuotaExhausted`` so callers - can honor the server's hint instead of guessing a wait.""" + can apply the server's value instead of guessing a wait.""" state = {"i": 0} async def fetch(args): @@ -1227,8 +1223,8 @@ async def fetch(args): def test_quota_exhausted_message_points_at_resume(): - """The error message must surface the chunk offset and the resume - affordance — ``partial_frame`` is a footgun without it.""" + """The error message must include the chunk offset and the resume + call — ``partial_frame`` is easy to misuse without it.""" e = QuotaExhausted( completed_chunks=7, total_chunks=20, @@ -1242,9 +1238,9 @@ def test_quota_exhausted_message_points_at_resume(): def test_request_bytes_sums_url_and_content(): """``_request_bytes`` returns ``len(str(url)) + len(content)``. - ``httpx.Request`` always carries ``.content`` as ``bytes`` (the + ``httpx.Request`` always has ``.content`` as ``bytes`` (the constructor normalises ``data``/``json``/``content`` inputs), so - the chunker just needs to size that single attribute alongside + the chunker needs only to size that single attribute alongside the URL. """ # GET request with no body @@ -1260,7 +1256,7 @@ def test_safe_request_bytes_treats_invalid_url_as_overflow(): """``httpx.URL`` enforces a 64 KB cap per URL component and raises ``httpx.InvalidURL`` for anything bigger — e.g. comma-joining all California stream sites in one query. The planner's halving loop - must keep shrinking past that cap rather than crashing; the + must keep shrinking past that cap rather than raising; the contract is that ``_safe_request_bytes`` returns ``url_limit + 1`` (a value strictly greater than the limit) when ``build_request`` raises ``InvalidURL``.""" @@ -1275,7 +1271,7 @@ def build_request(**kwargs): def test_chunk_plan_handles_initial_url_overflow(): """A user query whose unchunked URL exceeds the 64 KB ``httpx.URL`` cap (e.g. 5000+ site IDs comma-joined) must not - crash ``ChunkPlan.__init__``; the planner falls back to a + make ``ChunkPlan.__init__`` raise; the planner falls back to a worst-case chunk URL for ``canonical_url`` and proceeds to halve the over-limit axes normally.""" real_build = _fake_build @@ -1307,7 +1303,7 @@ def test_multi_value_chunked_restores_canonical_url(): @multi_value_chunked(build_request=_fake_build, url_limit=240) async def fetch(args): - # Each sub-response carries the chunked chunk_args's URL, so + # Each sub-response has the chunked chunk_args's URL, so # without canonical restoration the first chunk's URL would # leak through to md.url. sub_url = _fake_build(**args).url @@ -1322,14 +1318,14 @@ async def fetch(args): assert len(sub_urls) > 1, "test setup error: chunker didn't fan out" # md.url must equal the URL the unchunked query would have produced. assert md.url == _fake_build(sites=sites).url - # And differ from every chunk's URL (each carries a smaller list). + # And differ from every chunk's URL (each has a smaller list). assert all(md.url != u for u in sub_urls) # The canonical URL is strictly bigger byte-wise than any chunk. assert all(len(md.url) > len(u) for u in sub_urls) def test_extract_axes_skips_filter_passed_as_list(): - """Defensive guard: ``filter`` is documented as a string. If a caller + """Guard: ``filter`` is documented as a string. If a caller mistakenly passes it as a list, ``_extract_axes`` must NOT create a comma-joined list axis for it — comma-joining CQL clauses inside the URL would produce a malformed filter expression. The filter @@ -1345,10 +1341,10 @@ def test_extract_axes_skips_filter_passed_as_list(): def test_extract_axes_skips_scalar_contract_params(): """``limit`` and ``skip_geometry`` are scalars by contract - (``int | None`` and ``bool | None`` respectively). If a caller smuggles - a list through type erasure (e.g. ``limit=["100","200"]`` after an + (``int | None`` and ``bool | None`` respectively). If a caller passes + a list despite the annotation (e.g. ``limit=["100","200"]`` after an incorrect cast), ``_extract_axes`` must NOT treat it as a multi-value - axis. Chunking ``limit`` would silently fan into separate + axis. Chunking ``limit`` would fan into separate paginated queries with different per-request caps; chunking ``skip_geometry`` would emit chunks with conflicting geometry-output settings.""" @@ -1362,16 +1358,16 @@ def test_extract_axes_skips_scalar_contract_params(): def test_joint_planner_url_construction_long_filter_and_long_sites(): - """Realistic stress: 20 datetime OR-clauses combined with 100 USGS + """Stress test: 20 datetime OR-clauses combined with 100 USGS site IDs. Every chunk URL built from the plan must fit the 8000-byte limit, the joint planner must emit fewer chunks than splitting the filter to singletons and chunking the lists separately, and the partitioned filters must union to the user's original filter expression. Uses the real ``_construct_api_requests`` builder so the test - catches URL-encoding surprises that a fake builder would miss. + catches URL-encoding differences that a fake builder would miss. """ - # Realistic AGENCY-ID site format: USGS-{8 digits}. 500 sites is + # The AGENCY-ID site format: USGS-{8 digits}. 500 sites is # enough to force the URL well past the 8000-byte server limit # without any filter contribution. sites = [f"USGS-{i:08d}" for i in range(500)] @@ -1422,8 +1418,8 @@ def test_joint_planner_url_construction_long_filter_and_long_sites(): def test_combine_chunk_frames_all_empty_preserves_geo_type(): """An all-empty chunk list preserves the ``GeoDataFrame`` type. - Dropping empties before concat exists precisely to prevent type - downgrade; the all-empty branch must honor the same contract.""" + Dropping empties before concat exists to prevent type + downgrade; the all-empty branch must keep the same contract.""" pytest.importorskip("geopandas") import geopandas as gpd @@ -1446,7 +1442,7 @@ def test_combine_chunk_frames_single_frame_is_safe_to_mutate(): def test_iter_chunk_args_passthrough_yields_a_copy(): - """``ChunkPlan.iter_chunk_args`` yields a fresh dict on every path + """``ChunkPlan.iter_chunk_args`` yields a new dict on every path (passthrough and chunked), so a ``fetch_once`` that mutates the dict it receives cannot corrupt ``ChunkPlan.args``.""" args = {"monitoring_location_id": ["USGS-A"], "limit": 100} @@ -1491,7 +1487,7 @@ def _ok_response(remaining=None): def test_async_fan_out_emits_one_call_per_chunk(monkeypatch): - """The fan-out hits every sub-args exactly once, dispatched + """The fan-out calls every sub-args exactly once, dispatched concurrently.""" seen_args = [] @@ -1537,13 +1533,12 @@ def test_async_fan_out_failure_yields_resumable_call(monkeypatch): ``.call`` is a ``ChunkedCall`` holding the completed chunks in a sparse index map. ``exc.call.resume()`` re-issues only the unfinished chunks — through the same async fetcher and the same - async runner, just on a fresh gather.""" + async runner, only on a new gather.""" # One async fetcher serves both first-run and resume. On the first # gather it lets exactly one chunk succeed and fails the rest # transiently; once ``blow_up`` is cleared the resume gather completes # every still-pending chunk. ``calls`` counts every invocation - # across both gathers so we can assert resume only re-issued the owed - # chunks. + # across both gathers so we can assert resume only re-issued the incomplete chunks. state = {"first_success": False, "blow_up": True} calls = {"n": 0} @@ -1566,7 +1561,7 @@ async def fetch_async(args): interrupted = exc_info.value assert interrupted.call is not None, "interruption must be resumable" - # Exactly one chunk completed; the rest still owe. + # Exactly one chunk completed; the rest are still incomplete. assert interrupted.completed_chunks == 1 assert interrupted.total_chunks > 1 @@ -1582,10 +1577,10 @@ async def fetch_async(args): def test_async_fan_out_resume_applies_finalize(monkeypatch): - """The ``finalize`` injected for a wide-pool call survives the - interruption (carried on the ``ChunkedCall`` through the anyio portal), - so ``exc.call.resume()`` still returns the finalized shape — guarding - the run -> resume -> finalize path. Partials stay raw (no finalize in + """The ``finalize`` injected for a wide-pool call is kept across the interruption + (held on the ``ChunkedCall`` through the anyio portal), so ``exc.call.resume()`` + still returns the finalized shape — guarding the run -> resume -> finalize path. + Partials stay raw (no finalize in the exception ctor).""" def finalize(frame, response): @@ -1611,7 +1606,7 @@ async def fetch_async(args): # Partial snapshot stays raw — building the exception must not finalize. assert "finalized" not in exc_info.value.partial_frame.columns - # Resume applies the finalize carried on the ChunkedCall. + # Resume applies the finalize held on the ChunkedCall. state["blow_up"] = False df, md = exc_info.value.call.resume() assert "finalized" in df.columns @@ -1619,7 +1614,7 @@ async def fetch_async(args): def test_wide_concurrency_uses_async_fetcher_with_no_warning(monkeypatch): - """A wide ``API_USGS_CONCURRENT`` is honored directly by the single + """A wide ``API_USGS_CONCURRENT`` is applied directly by the single async fetcher: every chunk fans out across it and NO ``UserWarning`` is emitted.""" calls = [] @@ -1639,7 +1634,7 @@ async def fetch(args): # Eight 20-char sites against ``url_limit=240`` (base 200): any two atoms -# joined overflow the 40-byte budget, so the planner lands on eight +# joined overflow the 40-byte budget, so the planner produces eight # singleton chunks — enough fan-out to observe the concurrency gate. _EIGHT_SINGLETON_SITES = [f"S{i}" * 10 for i in range(8)] @@ -1670,14 +1665,14 @@ def test_fan_out_in_flight_high_water_mark_is_the_cap( monkeypatch, cap, expected_high_water ): """The fetch-level high-water mark of simultaneous chunks IS the - ``API_USGS_CONCURRENT`` cap — genuine parallelism up to it, never past - it — and ``unbounded`` degenerates to every chunk at once. + ``API_USGS_CONCURRENT`` cap — parallelism up to it, never past + it — and ``unbounded`` means every chunk at once. Regression: the cap used to be enforced only by the shared client's connection-pool size, so chunks beyond it queued on connection *acquisition*, subject to the client's pool-acquire timeout (see - ``ChunkedCall._run``). The semaphore parks excess chunks before - they touch the pool. + ``ChunkedCall._run``). The semaphore holds excess chunks back before + they reach the pool. """ in_flight = {"now": 0, "max": 0} fetch = _async_chunked_fetch( @@ -1706,21 +1701,18 @@ def test_configure_concurrency_controls_dispatch(monkeypatch): def test_fan_out_outlives_pool_timeout_on_real_transport(monkeypatch): - """End-to-end regression for the pool-timeout starvation bug: the - fan-out must survive every pooled connection staying busy past the - client's pool-acquire timeout (the stall mechanism is documented on - ``ChunkedCall._run``; at production scale think a batch of large, - slowly-streaming pages). - - Chunks here send real HTTP to a slow localhost server through - the chunker's shared client — fakes can't catch this, since - ``MockTransport`` bypasses the connection pool. With the pool as the - only throttle, 2 connections busy for 0.35 s each and the 0.2 s pool - timeout pinned below, the 2 queued chunks sat out the full - timeout with no completion to reset their clocks → - ``httpx.PoolTimeout`` → (retries exhausted, ``API_USGS_RETRIES=0``) - a spurious resumable ``ServiceInterrupted``. Gated by the semaphore, - queued chunks never touch the pool and the call completes. + """End-to-end regression for the pool-timeout starvation bug: the fan-out must + complete even when every pooled connection stays busy past the client's pool-acquire + timeout (the stall mechanism is documented on ``ChunkedCall._run``; at production + scale, a batch of large, slowly-streaming pages). + + Chunks here send real HTTP to a slow localhost server through the chunker's shared + client — fakes can't catch this, since ``MockTransport`` bypasses the connection + pool. With the pool as the only throttle, 2 connections busy for 0.35 s each and the + 0.2 s pool timeout pinned below, the 2 queued chunks waited the full timeout with no + completion to reset their timers → ``httpx.PoolTimeout`` → (retries exhausted, + ``API_USGS_RETRIES=0``) a spurious resumable ``ServiceInterrupted``. Gated by the + semaphore, queued chunks never reach the pool and the call completes. """ class _SlowHandler(http.server.BaseHTTPRequestHandler): @@ -1734,7 +1726,7 @@ def do_GET(self): self.end_headers() self.wfile.write(body) - def log_message(self, *args): # silence the server's request log + def log_message(self, *args): # suppress the server's request log pass server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _SlowHandler) @@ -1782,7 +1774,7 @@ async def fetch(args): # the single async fetcher return pd.DataFrame({"id": [_atom_id(args)]}), _ok_response() async def driver(): # call the sync getter from within a running loop - # The sync wrapper drives the async core through the anyio portal in + # The sync wrapper runs the async core through the anyio portal in # a worker thread, so it works even inside a running event loop # without raising a nested-``asyncio.run`` error. return fetch({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) @@ -1798,18 +1790,17 @@ def test_async_fan_out_cancellation_wins_over_transient_sibling(monkeypatch): transient (which would otherwise wrap as a resumable :class:`ChunkInterrupted`). Cancellation is asyncio's abort signal — letting a transient-classification path consume it - would silently swallow the user's stop request. + would suppress the user's stop request. ``fetch_async`` has no ``await`` in its body, so the gather schedules the tasks in submission order and each runs synchronously to its raise — making ``call_count`` deterministic: 1 = first chunk (success), 2 = second chunk (transient), 3 = third chunk (cancel). - Through the sync→async blocking portal an in-flight cancellation - surfaces to the caller as ``concurrent.futures.CancelledError`` (the - thread-boundary cancellation type) rather than ``asyncio.CancelledError`` - — either way it propagates unmodified rather than being swallowed and - wrapped as a resumable ``ChunkInterrupted``. + Through the sync→async blocking portal an in-flight cancellation reaches the caller + as ``concurrent.futures.CancelledError`` (the thread-boundary cancellation type) + rather than ``asyncio.CancelledError`` — either way it propagates unmodified rather + than being suppressed and wrapped as a resumable ``ChunkInterrupted``. """ call_count = {"async": 0} @@ -1837,12 +1828,11 @@ async def fetch_async(args): def test_combine_chunk_responses_does_not_mutate_input_urls(): """Regression for the _set_response_url aliasing bug. - ``_combine_chunk_responses`` shallow-copies the first response; - if the canonical-URL override is applied by mutating the bound - ``request.url``, the shallow alias back-propagates the URL change - into the underlying chunk-0 response — breaking the documented - 'input responses are not mutated' invariant. The fix is to swap - in a fresh ``httpx.Request`` rather than mutate the existing one. + ``_combine_chunk_responses`` shallow-copies the first response; if the canonical-URL + override is applied by mutating the bound ``request.url``, the shallow alias + back-propagates the URL change into the underlying chunk-0 response — breaking the + documented 'input responses are not mutated' invariant. The fix is to swap in a new + ``httpx.Request`` rather than mutate the existing one. """ req1 = httpx.Request("GET", "https://example.com/chunk0") req2 = httpx.Request("GET", "https://example.com/chunk1") @@ -1853,7 +1843,7 @@ def test_combine_chunk_responses_does_not_mutate_input_urls(): [r1, r2], canonical_url="https://canonical.example/full" ) assert str(out.url) == "https://canonical.example/full" - # The inputs and their bound requests must be untouched. + # The inputs and their bound requests must be unchanged. assert str(r1.url) == "https://example.com/chunk0" assert str(r2.url) == "https://example.com/chunk1" assert str(req1.url) == "https://example.com/chunk0" @@ -1861,7 +1851,7 @@ def test_combine_chunk_responses_does_not_mutate_input_urls(): # --------------------------------------------------------------------------- -# Retry-with-backoff: RetryPolicy + _retryable + driver + decorator wiring. +# Retry-with-backoff: RetryPolicy + _retryable + driver + decorator composition. # Conftest pins API_USGS_RETRIES=0, so these tests opt in explicitly and # patch the chunking module's ``asyncio.sleep`` to a no-op (no real backoff). # --------------------------------------------------------------------------- @@ -1885,7 +1875,7 @@ def test_retry_policy_backoff_full_jitter_within_ceiling(): for attempt, ceiling in [(1, 2.0), (2, 4.0), (3, 8.0), (5, 30.0)]: samples = [policy.backoff(attempt, None) for _ in range(200)] assert all(0.0 <= s <= ceiling for s in samples) - # Full jitter genuinely varies and reaches below the ceiling. + # Full jitter varies and falls below the ceiling. assert min(samples) < ceiling @@ -1898,7 +1888,7 @@ def test_retry_policy_should_retry_exhaustion(): def test_retry_policy_long_retry_after_escalates(): policy = RetryPolicy(max_retries=5, retry_after_cap=60.0) - assert policy.should_retry(attempt=1, retry_after=30.0) # absorbed inline + assert policy.should_retry(attempt=1, retry_after=30.0) # waited out inline assert not policy.should_retry(attempt=1, retry_after=120.0) # escalates @@ -1929,8 +1919,8 @@ def test_retry_policy_rejects_invalid_settings(): def test_retry_policy_from_config_honors_monkeypatched_constants(monkeypatch): - # The timing knobs are read from the module constants at call time, so - # monkeypatching them (as the module comment promises) takes effect. + # The timing values are read from the module constants at call time, so + # monkeypatching them (as the module comment states) takes effect. monkeypatch.setattr(_retry_mod, "_RETRY_MAX_BACKOFF", 0.0) monkeypatch.setattr(_retry_mod, "_RETRY_BASE_BACKOFF", 0.0) policy = RetryPolicy.from_configuration() @@ -1957,7 +1947,7 @@ def test_retryable_taxonomy(): def test_retryable_skips_wrapped_midpagination_transient(): - # A transient surfaced mid-pagination is re-wrapped as DataRetrievalError by + # A transient raised mid-pagination is re-wrapped as DataRetrievalError by # _paginate; it must NOT be auto-retried (re-walking from page 1 # would re-spend quota) — it escalates to the resumable handle instead. # Only the raw, top-level (initial-request) transient is retryable. @@ -1965,13 +1955,13 @@ def test_retryable_skips_wrapped_midpagination_transient(): assert _retryable(RateLimited("429", retry_after=3.0)) == (True, 3.0) -# -- async driver (the single retry driver; sync facade drives it) ---------- +# -- async driver (the single retry driver; the sync facade calls it) -------- # -# The retry loop lives in ``_retry``. These tests pin its behavioral +# The retry loop is ``_retry``. These tests pin its behavioral # contracts (transient-then-success, exhausted-reraises, # non-retryable-not-retried, long-retry-after-escalates), run via # ``asyncio.run``; the sleep is patched to a no-op so backoff doesn't -# actually wait. +# wait. def test_retry_transient_then_recovers(monkeypatch): @@ -2053,7 +2043,7 @@ async def afn(): def test_chunker_retries_transient_then_completes(monkeypatch): - """A transient on one chunk is retried transparently; the + """A transient on one chunk is retried automatically; the decorated call completes with no ChunkInterrupted.""" monkeypatch.setenv("API_USGS_RETRIES", "3") monkeypatch.setattr(_retry_mod.asyncio, "sleep", _aiozero) @@ -2092,8 +2082,8 @@ async def fetch(args): def test_chunker_exhausted_retries_still_resumable(monkeypatch): - """When retries are exhausted the failure still surfaces as a - resumable ChunkInterrupted — retries don't swallow the escape hatch.""" + """When retries are exhausted the failure is still raised as a + resumable ChunkInterrupted — retries do not remove the resume handle.""" monkeypatch.setenv("API_USGS_RETRIES", "2") monkeypatch.setattr(_retry_mod.asyncio, "sleep", _aiozero) attempts = {"n": 0} @@ -2109,7 +2099,7 @@ async def fetch(args): with pytest.raises(ServiceInterrupted) as excinfo: decorated({"sites": ["S1" * 10, "S2" * 10, "S3" * 10, "S4" * 10]}) assert excinfo.value.call is not None - assert attempts["n"] == 3 # first attempt + 2 retries before giving up + assert attempts["n"] == 3 # first attempt + 2 retries before stopping def test_async_fan_out_retries_transient_then_completes(monkeypatch): @@ -2131,15 +2121,15 @@ async def fetch_async(args): def test_async_fan_out_surfaces_fatal_over_transient(monkeypatch): - """A non-transient bug in one chunk surfaces raw rather than + """A non-transient bug in one chunk is raised unwrapped rather than being masked behind a resumable interruption from a transient sibling.""" monkeypatch.setenv("API_USGS_RETRIES", "2") monkeypatch.setattr(_retry_mod.asyncio, "sleep", _aiozero) async def fetch_async(args): - # One chunk carries a deterministic programmer error; the rest are - # transient. The real bug must win over the resumable transient. + # One chunk raises a deterministic programmer error; the rest are transient. The + # programming error must take precedence over the resumable transient. if "S1" * 10 in args["sites"]: raise ValueError("deterministic bug") raise ServiceUnavailable("503: transient") @@ -2187,7 +2177,7 @@ async def fetch(_args): # terminal resume()/resume_async() returns. The partial_* accessors stay RAW # so building/inspecting a ChunkInterrupted never triggers finalize's side # effects (for OGC, _deal_with_empty issues a schema network GET on an empty -# frame — that must NOT fire inside the exception constructor). +# frame — that must NOT run inside the exception constructor). def test_resume_finalizes_but_partials_stay_raw(monkeypatch): @@ -2204,7 +2194,7 @@ def finalize(frame, response): # Fail the 2nd issued chunk once (the 1st completes, so partial # state is non-empty), then succeed on resume. Conftest pins a single - # connection and no retries, so the failure surfaces immediately. + # connection and no retries, so the failure is raised immediately. state = {"n": 0} @multi_value_chunked(build_request=_fake_build, url_limit=240) @@ -2234,20 +2224,20 @@ async def fetch(args): # --------------------------------------------------------------------------- -# Parallel chunks: the opt-in dial ``parallel_chunks(n)`` to fan a query out +# Parallel chunks: the opt-in setting ``parallel_chunks(n)`` to fan a query out # MORE finely than the byte limit alone requires (``ChunkPlan._refine`` + the # ``parallel_chunks`` context manager). ``_fake_build``'s base is 200 bytes, so -# a handful of short atoms sits far under ``url_limit=8000`` — the byte pass -# passes it through untouched, and any splitting below is the ``n`` cap alone. +# a few short atoms are far under ``url_limit=8000`` — the byte pass +# passes it through unchanged, and any splitting below is the ``n`` cap alone. # ``ChunkPlan`` takes the integer cap (``max_chunks``) directly; -# ``parallel_chunks(n)`` publishes ``n`` onto it. The cap bounds the plan's +# ``parallel_chunks(n)`` sets ``n`` on it. The cap bounds the plan's # *total* chunk count (the cartesian product across axes), not each axis # independently — see ``test_cap_caps_the_total_across_axes``. # --------------------------------------------------------------------------- def test_default_preserves_passthrough(): - """The default ``max_chunks`` (1 = off) must not perturb the existing + """The default ``max_chunks`` (1 = off) must not change the existing plan: a multi-value request that fits the byte limit is still the single-chunk passthrough (no axes, ``total == 1``), byte-for-byte the pre-feature behavior.""" @@ -2273,7 +2263,7 @@ def test_unit_cap_preserves_passthrough(): @pytest.mark.parametrize("bad", [0, -1]) def test_invalid_cap_raises(bad): """``max_chunks`` is a chunk count, so a value below 1 (``0`` or - negative) is a caller bug, not a silent no-op: it raises ``ValueError`` at + negative) is a caller bug, not an ignored no-op: it raises ``ValueError`` at construction. (The public ``parallel_chunks(n)`` already rejects ``n < 1``; this pins the same guard on direct construction.)""" args = {"monitoring_location_id": ["A", "B", "C", "D"]} @@ -2288,7 +2278,7 @@ def test_invalid_cap_raises(bad): def test_cap_ramps_then_saturates(max_chunks, expected_pieces): """A single 10-atom axis that fits the byte limit splits into ``min(10, cap)`` pieces: 1 (off), 2, 8, then saturating at 10 (one atom per - chunk) once the cap overshoots the atom count. Monotonic and bounded, and + chunk) once the cap exceeds the atom count. Monotonic and bounded, and whenever it splits the partition is a cover — every atom exactly once. (The cap-1 passthrough has no axis to cover; see the passthrough test.)""" atoms = [f"S{i:02d}" for i in range(10)] @@ -2309,7 +2299,7 @@ def test_cap_ramps_then_saturates(max_chunks, expected_pieces): def test_cap_bounds_fan_out_for_a_long_axis(): """The cap holds fan-out to ``n``: at ``n=32`` a 100-atom axis fans into ``n`` pieces — NOT 100 singletons — so ``parallel_chunks(32)`` on a huge - list can't detonate into hundreds of chunks. Every atom is still + list cannot expand into hundreds of chunks. Every atom is still covered exactly once.""" high = 32 atoms = [f"X{i:03d}" for i in range(100)] @@ -2328,8 +2318,8 @@ def test_cap_below_byte_split_does_not_reduce_fan_out(): """The cap is purely additive — it can only split further, never coarsen. A request the byte budget already fans into K>2 chunks is untouched by a cap of 2 (below K), so the byte-driven plan is preserved.""" - # Heavy axis of four 30-char atoms; a limit tight enough that the byte pass - # must drive every atom into its own chunk (4 pieces > the cap of 2). + # An axis of four 30-char atoms; a limit tight enough that the byte pass + # must put every atom into its own chunk (4 pieces > the cap of 2). args = {"monitoring_location_id": ["X" * 30, "Y" * 30, "Z" * 30, "W" * 30]} baseline = ChunkPlan(args, _fake_build, url_limit=250, max_chunks=1) assert baseline.total > 2 # byte pass alone already fanned out past 2 @@ -2352,7 +2342,7 @@ def test_cap_never_exceeds_the_byte_budget(): def test_cap_refines_the_filter_axis(): - """The dial treats the cql-text ``filter`` axis like any other: an + """The cap treats the cql-text ``filter`` axis like any other: an under-budget filter of N top-level OR-clauses is split along that axis into ``min(N, cap)`` pieces.""" clauses = [f"p='{i}'" for i in range(8)] @@ -2365,9 +2355,9 @@ def test_cap_refines_the_filter_axis(): def test_cap_caps_the_total_across_axes(): """With more than one multi-value axis the cap bounds the *total* chunk count (the cartesian product), not each axis independently — - the blast-radius guardrail the dial exists for. Two 6-atom axes at a cap - of 4 top out at 4 chunks total, not 4x4=16; growth is distributed - round-robin across axes rather than one axis alone climbing to the cap.""" + the bound on total chunk count the cap exists for. Two 6-atom axes at a cap + of 4 reach at most 4 chunks total, not 4x4=16; growth is distributed + round-robin across axes rather than one axis alone reaching the cap.""" args = { "monitoring_location_id": [f"L{i}" for i in range(6)], "parameter_code": [f"{i:05d}" for i in range(6)], @@ -2384,7 +2374,7 @@ def test_cap_caps_the_total_across_axes(): def test_cap_bounds_fan_out_across_many_axes(): - """The guardrail holds regardless of axis count: three multi-value axes at + """The bound holds regardless of axis count: three multi-value axes at a cap of 30 fan out to *at most* 30 chunks total — never the ``30 ** 3`` a per-axis cap would allow, and never *over* the cap either. 30 is deliberately not evenly reachable by these axes: a single split @@ -2406,19 +2396,20 @@ def test_cap_bounds_fan_out_across_many_axes(): @pytest.mark.parametrize( "atoms_per_axis, cap", [ - (4, 5), # pre-fix loop overshot 5 -> 6 + (4, 5), # pre-fix loop exceeded 5 -> 6 (8, 10), # pre-fix loop overshot 10 -> 12 (10, 7), # pre-fix loop overshot 7 -> 8 ], ) def test_cap_is_a_hard_ceiling_never_overshoots(atoms_per_axis, cap): """The cap is a hard ceiling, not a soft target. With two multi-value axes - a single split multiplies the plan by ``(k+1)/k`` for the split axis — - adding the product of the *other* axes, not one — so a - ``while total < cap`` loop steps *past* the cap. These are exactly the - (atoms, cap) combos that loop overshot (5->6, 10->12, 7->8). The plan must - fan out and cover every atom once, but never exceed the cap, landing below - it when no whole split lands on it exactly (two even axes reach 4, not 5).""" + a single split multiplies the plan by ``(k+1)/k`` for the split axis — adding + the product of the *other* axes, not one — so a ``while total < cap`` loop + steps past the cap. These are the (atoms, cap) combos on which that loop + exceeded the cap (5->6, 10->12, 7->8). The plan must fan out and cover every + atom once, but never exceed the cap, ending below it when no whole split + reaches it exactly (two even axes reach 4, not 5). + """ args = { "monitoring_location_id": [f"L{i:03d}" for i in range(atoms_per_axis)], "parameter_code": [f"{i:05d}" for i in range(atoms_per_axis)], @@ -2432,9 +2423,9 @@ def test_cap_is_a_hard_ceiling_never_overshoots(atoms_per_axis, cap): def test_cap_does_not_mask_unchunkable(): - """A request with nothing to split that still busts the byte limit must + """A request with nothing to split that still exceeds the byte limit must raise ``Unchunkable`` regardless of the cap — the soft pass has no axis to - act on and must not swallow the raise.""" + act on and must not suppress the raise.""" args = {"monitoring_location_id": "one-huge-scalar"} with pytest.raises(Unchunkable): ChunkPlan(args, _fake_build, url_limit=10, max_chunks=32) @@ -2445,9 +2436,9 @@ def test_parallel_chunks_publishes_n_as_the_effective_setting(): value on exit — including across nested blocks. ``parallel_chunks(n)`` is sugar for ``configure(parallel_chunks=n)``, so - both forms share one scoping mechanism and the innermost block wins. + both forms share one scoping mechanism and the innermost block takes precedence. Outside any block the configured baseline applies, which is ``1`` — off — - unless a config file raised it.""" + unless a config file set it higher.""" assert _configuration.parallel_chunks() == 1 # default (off, = no extra fan-out) with parallel_chunks(32): assert _configuration.parallel_chunks() == 32 @@ -2505,7 +2496,7 @@ async def fetch(args): calls.append(chunk) return pd.DataFrame({"site": list(chunk)}), _ok_response() - # Default: comfortably under the byte limit → one passthrough call. + # Default: well under the byte limit → one passthrough call. df_plain, _ = fetch({"monitoring_location_id": sites}) assert len(calls) == 1 assert sorted(df_plain["site"]) == sorted(sites) @@ -2541,7 +2532,7 @@ async def fetch(args): class TestSetResponseUrl: - """The combined response advertises the canonical URL, not the last + """The combined response reports the canonical URL, not the last chunk's. ``httpx.Response`` resolves ``.url`` through its bound request, so the rebind has to go through the request rather than the attribute.""" @@ -2556,8 +2547,9 @@ def test_a_response_with_no_bound_request_gets_one_synthesized(self): assert str(response.url) == "https://example.test/combined" def test_an_existing_request_keeps_its_method_and_headers(self): - """A combined POST must not silently become a GET, and the headers - carry the credential scoping.""" + """A combined POST must not become a GET, and the headers hold the credential + scoping. + """ from dataretrieval.combining import _set_response_url original = httpx.Request( diff --git a/tests/waterdata_filters_test.py b/tests/waterdata_filters_test.py index 1272602a7..1f3bfee8d 100644 --- a/tests/waterdata_filters_test.py +++ b/tests/waterdata_filters_test.py @@ -32,7 +32,7 @@ def _query_params(prepared_request): def _fake_prepared_request(url="https://example.test"): """Stand-in for the object ``_construct_api_requests`` returns. - Carries ``content`` because the planner sizes candidate chunks as + Has ``content`` because the planner sizes candidate chunks as ``len(url) + len(content)`` — and with the builder bound per call, a patched builder is what the planner measures. """ @@ -49,8 +49,9 @@ def _fake_response(url="https://example.test", elapsed_ms=1): def test_quote_cql_str_doubles_embedded_quotes(): - """The shared CQL-text escaper doubles ``'`` and leaves other input - untouched (the contract ``waterdata.ratings._build_filter`` relies on).""" + """The shared CQL-text escaper doubles ``'`` and leaves other input unchanged (the + contract ``waterdata.ratings._build_filter`` relies on). + """ assert _quote_cql_str("O'Brien") == "O''Brien" assert _quote_cql_str("USGS-01646500") == "USGS-01646500" assert _quote_cql_str("a'b'c") == "a''b''c" @@ -91,8 +92,8 @@ def test_split_top_level_or_respects_quotes(): def test_split_top_level_or_handles_doubled_quote_escape(): """CQL text escapes a single quote inside a literal as ``''``. The two quotes are adjacent, so the scanner's escape-unaware toggle-on-quote - logic happens to land back in the correct state with nothing between the - toggles to misclassify. Lock that behavior in so a future refactor + logic happens to return to the correct state with nothing between the + toggles to misclassify. Pin that behavior so a future refactor can't regress it.""" cases = [ ("name = 'O''Reilly OR Co' OR id = 1", ["name = 'O''Reilly OR Co'", "id = 1"]), @@ -149,7 +150,7 @@ def _filter_chunking_clauses(n: int = 300) -> str: def _filter_size_aware_build(**kwargs): """Fake ``_construct_api_requests`` whose returned URL length scales - with the request's ``filter`` value, so the joint planner naturally + with the request's ``filter`` value, so the joint planner triggers chunking on long filters.""" return _fake_prepared_request( url=f"https://example.test/?filter={kwargs.get('filter', '')}", @@ -307,9 +308,9 @@ def fake_construct_api_requests(**kwargs): filter_lang="cql-json", ) - # The planner sizes through the (patched) builder and the fetch builds - # through it again; every call must carry the caller's cql-json filter - # verbatim — never split — and exactly one chunk is fetched. + # The planner sizes through the (patched) builder and the fetch builds through it + # again; every call must include the caller's cql-json filter verbatim — never split + # — and exactly one chunk is fetched. assert sent_filters and set(sent_filters) == {expr} assert walk.await_count == 1 @@ -327,7 +328,7 @@ def fake_construct_api_requests(**kwargs): "value >= 1000.5", "value >= -50", # Zero-padded codes: `parameter_code = 60` matches nothing - # because the real values are all `'00060'`-shaped + # because the real values are all zero-padded like `'00060'` "parameter_code = 60", "statistic_id = 11", "district_code = 1", @@ -344,14 +345,14 @@ def fake_construct_api_requests(**kwargs): "value > .5", "value >= -.5", "value < .5e-3", - # ``IN`` list form — same footgun, common pattern for codes + # ``IN`` list form — same pitfall, common pattern for codes "parameter_code IN (60, 61)", "value IN (10, 20, 30)", "statistic_id in (11)", # case-insensitive, single-element - # ``NOT IN`` with numbers — same footgun via negation + # ``NOT IN`` with numbers — same pitfall via negation "value NOT IN (1, 2, 3)", "parameter_code not in (60, 61)", - # ``BETWEEN`` range form — same footgun + # ``BETWEEN`` range form — same pitfall "value BETWEEN 5 AND 10", "channel_flow between 100 and 500", # ``NOT BETWEEN`` with numbers @@ -390,7 +391,7 @@ def test_check_numeric_filter_pitfall_raises(expr): "qualifier IN ('A', 'P')", "parameter_code IN ('00060', '00065')", "value BETWEEN '1' AND '9'", - # Footgun identifiers appearing only inside string literals + # Pitfall identifiers appearing only inside string literals "monitoring_location_id = 'USGS-value >= 1000'", "name = 'why I care about parameter_code = 60'", "note = 'see district_code = 1 in docs'", @@ -434,7 +435,7 @@ def test_pitfall_error_names_real_field_not_NOT_keyword(expr, field, op): def test_get_continuous_surfaces_pitfall_to_caller(): """End-to-end: the check runs at the ``get_continuous`` boundary, - not as a deep internal-only protection, so callers see the error + not only inside an internal helper, so callers get the error before any HTTP traffic.""" with mock.patch("dataretrieval.ogc.engine._construct_api_requests") as build: with pytest.raises(ValueError, match="lexicographic"): @@ -448,7 +449,7 @@ def test_get_continuous_surfaces_pitfall_to_caller(): class TestOrSeparatorBoundaries: - """Top-level ``OR`` splitting drives chunking, so a false split changes + """Top-level ``OR`` splitting determines chunking, so a false split changes the query's meaning and a missed one leaves an unchunkable filter.""" def test_a_word_merely_starting_with_or_is_not_a_separator(self): diff --git a/tests/waterdata_nearest_test.py b/tests/waterdata_nearest_test.py index cca36f6b4..45d834184 100644 --- a/tests/waterdata_nearest_test.py +++ b/tests/waterdata_nearest_test.py @@ -1,7 +1,7 @@ """Tests for ``waterdata.get_nearest_continuous``. All network interaction is mocked at the ``get_continuous`` boundary, so -these run without an API key and without touching the USGS servers. +these run without an API key and without contacting the USGS servers. """ from unittest import mock @@ -15,7 +15,7 @@ def _fake_df(rows): - """Build a minimal continuous-response-shaped DataFrame.""" + """Build a minimal DataFrame with the continuous-response columns.""" return pd.DataFrame( { "time": pd.to_datetime([r["time"] for r in rows], utc=True), @@ -137,7 +137,7 @@ def test_tie_mean_averages_numeric_and_uses_target_time(patch_get_continuous): window="PT7M30S", ) assert result.iloc[0]["value"] == pytest.approx(22.2) - # Time is set to the target since no real observation sits at the midpoint + # Time is set to the target since no real observation falls at the midpoint assert result.iloc[0]["time"] == targets[0] @@ -174,9 +174,8 @@ def test_multi_site_returns_row_per_target_per_site(patch_get_continuous): def test_empty_targets_raises(patch_get_continuous): - """An empty ``targets`` is a call with no useful work to do and - almost always a caller bug — raise rather than silently issuing a - no-op HTTP request.""" + """An empty ``targets`` is a call with no useful work to do and almost always a + caller bug — raise rather than issue a request that returns nothing.""" with pytest.raises(ValueError, match="targets"): get_nearest_continuous([], monitoring_location_id="USGS-02238500") patch_get_continuous.assert_not_called() @@ -421,8 +420,8 @@ def test_caller_properties_keep_the_columns_the_match_needs(patch_get_continuous Without the injection a list like ``['time', 'value']`` reached the service unchanged, the response came back with no - ``monitoring_location_id``, and every site but one was silently dropped -- - an incomplete result with nothing for a caller to notice it by. + ``monitoring_location_id``, and every site but one was dropped without an error -- + an incomplete result the caller could not detect. """ patch_get_continuous.return_value = ( pd.DataFrame( @@ -506,9 +505,9 @@ def test_empty_response_does_not_require_matching_columns(patch_get_continuous): def test_no_observation_inside_the_window_returns_the_empty_shape( patch_get_continuous, ): - """A target with nothing near it is a legitimate answer, not a failure -- + """A target with nothing near it is a valid result, not a failure -- but the frame must keep the result columns so a caller can concatenate it - with a populated one instead of special-casing empties.""" + with a populated one instead of special-casing empty frames.""" patch_get_continuous.return_value = ( pd.DataFrame( [ diff --git a/tests/waterdata_progress_test.py b/tests/waterdata_progress_test.py index 7c0076eda..2286c81af 100644 --- a/tests/waterdata_progress_test.py +++ b/tests/waterdata_progress_test.py @@ -31,7 +31,7 @@ def _run_walk_pages(*, geopd, req, client): - """Drive the async ``_walk_pages`` to completion synchronously. + """Run the async ``_walk_pages`` to completion synchronously. The chunker core is async-only now, so these tests build an ``AsyncMock(spec=httpx.AsyncClient)`` whose ``.send``/``.request`` are @@ -42,7 +42,7 @@ def _run_walk_pages(*, geopd, req, client): return asyncio.run(_walk_pages(geopd=geopd, req=req, client=client)) -# The Water Data host is the only one that honors ``API_USGS_PAT``, and so the +# The Water Data host is the only one that accepts ``API_USGS_PAT``, and so the # only one where pointing the user at API-key registration is useful advice. _KEYED_URL = "https://api.waterdata.usgs.gov/ogcapi/v0/" @@ -132,7 +132,7 @@ def test_note_retry_is_noop_when_disabled(): def test_note_retry_accepts_integer_wait(): # An int ``wait`` (e.g. whole seconds) must render without raising: # ``round(int, 1)`` returns an int and ``int.is_integer()`` only exists on - # Python 3.12+, while the package floor is 3.10. Renders like the float. + # Python 3.12+, while the package minimum is 3.10. Renders like the float. stream = io.StringIO() reporter = ProgressReporter(stream=stream, enabled=True) reporter.note_retry(attempt=1, wait=5) @@ -216,8 +216,8 @@ def flush(self): def test_reporter_swallows_stream_errors_and_disables(monkeypatch): monkeypatch.delenv("API_USGS_PAT", raising=False) reporter = ProgressReporter(stream=_RaisingStream(), enabled=True) - reporter.add_page(rows=1) # render write raises -> must be swallowed - reporter.close() # newline + hint writes raise -> must be swallowed + reporter.add_page(rows=1) # render write raises -> must be caught + reporter.close() # newline + hint writes raise -> must be caught assert reporter.enabled is False @@ -234,8 +234,8 @@ def test_hints_api_key_when_no_key_configured(monkeypatch): def test_hint_fires_even_when_rate_limit_was_seen(monkeypatch): - # Anonymous responses still carry a rate-limit header, so absence of a key - # — not absence of the header — is what drives the pointer. + # Anonymous responses still include a rate-limit header, so absence of a key + # — not absence of the header — is what decides the pointer. monkeypatch.delenv("API_USGS_PAT", raising=False) stream = io.StringIO() reporter = ProgressReporter(stream=stream, enabled=True, target_url=_KEYED_URL) @@ -255,10 +255,10 @@ def test_no_hint_when_api_key_present(monkeypatch): def test_no_hint_for_a_service_the_key_does_not_cover(monkeypatch): - """Only the host that honors ``API_USGS_PAT`` gets the sign-up pointer. + """Only the host that accepts ``API_USGS_PAT`` gets the sign-up pointer. - Water Use is on a different host and never receives the key, so telling its - users to register sends them after a fix that changes nothing. + Water Use is on a different host and never receives the key, so telling its users to + register directs them to a change that has no effect. """ monkeypatch.delenv("API_USGS_PAT", raising=False) stream = io.StringIO() @@ -326,7 +326,7 @@ def _fake_ipython(shell_class_name): def test_enabled_in_jupyter_kernel(monkeypatch): # A Jupyter kernel's stderr isn't a TTY, but the line should still show - # (it honors \r in the cell output, like tqdm). + # (it handles \r in the cell output, like tqdm). monkeypatch.delenv("API_USGS_PROGRESS", raising=False) monkeypatch.setitem(sys.modules, "IPython", _fake_ipython("ZMQInteractiveShell")) assert ProgressReporter(stream=io.StringIO()).enabled is True @@ -373,7 +373,7 @@ def test_nested_context_reuses_outer_reporter(): def _resp(features, *, next_url=None, rate_remaining=None): resp = mock.MagicMock() - # A real response always carries an ``httpx.URL``; the next-page check + # A real response always has an ``httpx.URL``; the next-page check # resolves and host-checks the ``next`` link against it. resp.url = httpx.URL("https://example.com/p1") links = [{"rel": "next", "href": next_url}] if next_url else [] @@ -413,7 +413,8 @@ def test_walk_pages_reports_pages_and_rate_limit(): assert len(df) == 2 out = stream.getvalue() - # The collection set on the context reaches _paginate's render via the contextvar. + # The collection set on the context is available to _paginate's render via the + # contextvar. assert "Retrieving: daily ·" in out assert "2 pages" in out assert "4,998 requests remaining" in out @@ -421,7 +422,7 @@ def test_walk_pages_reports_pages_and_rate_limit(): def test_walk_pages_without_context_does_not_error(): - # No active reporter: pagination must still work and stay silent. + # No active reporter: pagination must still work and print nothing. resp = _resp([{"id": "1", "properties": {"v": "a"}}]) client = mock.AsyncMock(spec=httpx.AsyncClient) client.send.return_value = resp @@ -437,8 +438,8 @@ def test_walk_pages_without_context_does_not_error(): def test_broken_progress_stream_does_not_truncate_pagination(): - # A render failure (broken pipe) lands inside _walk_pages' per-page try; - # it must NOT be mistaken for a failed request and silently drop pages. + # A render failure (broken pipe) is raised inside _walk_pages' per-page try; it must + # not be mistaken for a failed request and drop pages without an error. resp1 = _resp( [{"id": "1", "properties": {"v": "a"}}], next_url="https://example.com/p2" ) @@ -462,10 +463,9 @@ def test_broken_progress_stream_does_not_truncate_pagination(): def test_paginate_reports_pages_through_active_reporter(monkeypatch): - """The async paginate path must drive the same progress reporter. - Pages and rate-limit updates from each completed page should land - via the active ``ProgressReporter``, exactly as they would on - ``_walk_pages``.""" + """The async paginate path must report through the same progress reporter. Pages and + rate-limit updates from each completed page should be reported through the active + ``ProgressReporter``, as they are on ``_walk_pages``.""" resp1 = _resp( [{"id": "1", "properties": {"v": "a"}}], next_url="https://example.com/p2", @@ -523,13 +523,12 @@ async def run(): def test_fan_out_async_sets_chunks_on_active_reporter(monkeypatch): """The async fan-out core (``ChunkedCall._run``) records - ``plan.total`` on the active reporter so the progress line knows how - many chunks are in flight, and ticks ``current_chunk`` via - ``start_chunk(len(completed))`` as each gathered chunk finishes - — reaching ``plan.total`` in the all-success case.""" + ``plan.total`` on the active reporter so the progress line has the count of chunks + in flight, and increments ``current_chunk`` via ``start_chunk(len(completed))`` as + each gathered chunk finishes — reaching ``plan.total`` in the all-success case.""" # Fake build_request whose URL length scales with the sites list, - # mirroring the planner's _request_bytes contract. _FakeReq has the + # matching the planner's _request_bytes contract. _FakeReq has the # same shape as httpx.Request for sizing purposes. class _FakeReq: __slots__ = ("url", "content") @@ -554,7 +553,7 @@ async def fetch_async(args): stream = io.StringIO() async def run(): - # Drive the async execution core directly (the same coroutine the + # Run the async execution core directly (the same coroutine the # sync ``resume()`` facade runs through the anyio portal). with progress_context(service="daily", stream=stream, enabled=True) as rep: await ChunkedCall(plan, fetch_async)._run(4) @@ -562,7 +561,7 @@ async def run(): total_recorded, current_recorded = asyncio.run(run()) assert total_recorded == plan.total - # Each chunk that completes bumps current_chunk via + # Each chunk that completes increments current_chunk via # start_chunk(len(completed)), so by the time the gather finishes # current_chunk reflects the total number of successful chunks — # plan.total in the all-success case. @@ -583,8 +582,8 @@ def test_closing_twice_is_a_no_op(): def test_a_broken_stream_disables_the_reporter_instead_of_failing_the_query(): - """Progress is decoration. A closed or redirected stream must not take - down a query whose data already arrived.""" + """Progress output is not part of the result. A closed or redirected + stream must not fail a query whose data already arrived.""" class _Broken(io.StringIO): def write(self, s): diff --git a/tests/waterdata_queryables_test.py b/tests/waterdata_queryables_test.py index 9494c18ab..a0b2c262f 100644 --- a/tests/waterdata_queryables_test.py +++ b/tests/waterdata_queryables_test.py @@ -1,11 +1,11 @@ """Tests for :func:`dataretrieval.waterdata.get_queryables`, plus a live monitor that flags upstream changes to the Water Data API's queryable sets. -The live monitor (:func:`test_queryables_match_snapshot`) compares the -queryables each collection advertises against a committed snapshot -(``tests/data/waterdata_queryables.json``). When it fails, the upstream API has -added / removed / renamed a queryable: regenerate the snapshot and enable any -new queryables on the matching getter. Regenerate with:: +The live monitor (:func:`test_queryables_match_snapshot`) compares the queryables each +collection publishes against a committed snapshot +(``tests/data/waterdata_queryables.json``). When it fails, the upstream API has added / +removed / renamed a queryable: regenerate the snapshot and enable any new queryables on +the matching getter. Regenerate with:: python - <<'PY' import httpx, json @@ -66,7 +66,7 @@ def test_get_queryables_parses_properties(httpx_mock): - """Properties become one tidy row each, sorted by name, with the + """Properties become one row each, sorted by name, with the description whitespace-stripped; returns ``(DataFrame, BaseMetadata)``.""" httpx_mock.add_response(method="GET", url=QUERYABLES_RE, json=_FAKE_QUERYABLES) @@ -83,7 +83,7 @@ def test_get_queryables_parses_properties(httpx_mock): def test_get_queryables_unknown_collection_raises(httpx_mock): - """An HTTP error (e.g. a 404 for an unknown collection) is surfaced as the + """An HTTP error (e.g. a 404 for an unknown collection) is raised as the typed ``DataRetrievalError``, not a bare DataFrame.""" httpx_mock.add_response( method="GET", @@ -114,7 +114,7 @@ def test_get_queryables_unknown_collection_raises(httpx_mock): def _mock_daily(httpx_mock): - """Mock the two endpoints a ``get_daily`` call touches: the items query and + """Mock the two endpoints a ``get_daily`` call requests: the items query and the schema fetch (used for output typing).""" httpx_mock.add_response(method="GET", url=_DAILY_SCHEMA_RE, json={"properties": {}}) httpx_mock.add_response(method="GET", url=_DAILY_ITEMS_RE, json=_EMPTY_FEATURES) diff --git a/tests/waterdata_ratings_test.py b/tests/waterdata_ratings_test.py index 598c5316f..eecabc9d4 100644 --- a/tests/waterdata_ratings_test.py +++ b/tests/waterdata_ratings_test.py @@ -10,10 +10,9 @@ from dataretrieval.waterdata import get_ratings from dataretrieval.waterdata.ratings import _build_filter -# pytest-httpx matches URL strings exactly (including query). For the -# ratings tests we want a "match this endpoint, ignore the params" -# fixture so the assertions can drill into the captured params -# afterwards without coupling the registration to the implementation's +# pytest-httpx matches URL strings exactly (including query). For the ratings tests the +# fixture matches the endpoint and ignores the params, so the assertions can inspect the +# captured params afterwards without coupling the registration to the implementation's # parameter order. ``url=STAC_SEARCH_RE`` does that. STAC_SEARCH_RE = re.compile( r"^https://api\.waterdata\.usgs\.gov/stac/v0/search(\?.*)?$" @@ -45,7 +44,7 @@ def test_get_ratings_rejects_invalid_file_type(): def test_get_ratings_rejects_iso_8601_duration_in_time(): - """STAC ratings doesn't accept ISO 8601 durations; surface a clear error.""" + """STAC ratings doesn't accept ISO 8601 durations; raise a clear error.""" with pytest.raises(ValueError, match=r"durations.*not supported"): get_ratings( monitoring_location_id="USGS-01104475", @@ -54,7 +53,7 @@ def test_get_ratings_rejects_iso_8601_duration_in_time(): def test_build_filter_escapes_quotes(): - """Defends against malformed CQL or injection if an ID contains a quote.""" + """Prevents malformed CQL or injection if an ID contains a quote.""" f = _build_filter("USGS-x'-y", None) assert f == "monitoring_location_id IN ('USGS-x''-y')" @@ -106,7 +105,7 @@ def test_get_ratings_mocked_search_and_download(httpx_mock, tmp_path): assert {"INDEP", "DEP"}.issubset(df.columns) assert len(df) == 3 - # Server-side filter should pin the single requested file_type. + # Server-side filter should set the single requested file_type. sent = httpx_mock.get_requests()[0] qs = parse_qs(urlsplit(str(sent.url)).query) assert "file_type = 'exsa'" in qs["filter"][0] @@ -114,7 +113,7 @@ def test_get_ratings_mocked_search_and_download(httpx_mock, tmp_path): def test_get_ratings_attaches_rdb_comment_and_url(httpx_mock, tmp_path): - """Each parsed frame should carry its RDB header + source URL in df.attrs.""" + """Each parsed frame should hold its RDB header and source URL in df.attrs.""" httpx_mock.add_response( method="GET", url=STAC_SEARCH_RE, @@ -128,7 +127,7 @@ def test_get_ratings_attaches_rdb_comment_and_url(httpx_mock, tmp_path): file_path=str(tmp_path), ) df = out["USGS-01104475.exsa.rdb"] - # The fixture has two `# ...` lines at the top; both should land in attrs. + # The fixture has two `# ...` lines at the top; both should appear in attrs. assert df.attrs["comment"] == [ "# header line one", "# header line two", @@ -197,8 +196,8 @@ def test_get_ratings_multi_type_filters_via_property(httpx_mock, tmp_path): def test_get_ratings_search_429_is_resumable(httpx_mock): - """A rate-limited search surfaces as a resumable interruption — parity - with the other getters, which drive the same executor — instead of a raw + """A rate-limited search raises a resumable interruption — the same as the other + getters, which use the same executor — instead of a raw ``RateLimited``; resuming finishes the interrupted stage.""" httpx_mock.add_response(method="GET", url=STAC_SEARCH_RE, status_code=429) httpx_mock.add_response( @@ -248,7 +247,7 @@ def test_get_ratings_deterministic_download_failure_warns_and_skips(httpx_mock): def test_get_ratings_feature_without_asset_warns_and_skips(httpx_mock): - """A catalog feature carrying no data asset is a per-feature data problem: + """A catalog feature with no data asset is a per-feature data problem: skipped with a warning, without costing the rest of the batch.""" body = _two_feature_search_response() body["features"][0]["assets"] = {} @@ -264,7 +263,7 @@ def test_get_ratings_feature_without_asset_warns_and_skips(httpx_mock): def test_get_ratings_skip_warning_escalates_to_error(httpx_mock): """``filterwarnings("error", ...)`` restores strict all-or-nothing: the - escalated skip surfaces as an exception instead of a silent gap.""" + escalated skip is raised as an exception instead of leaving an unreported gap.""" httpx_mock.add_response( method="GET", url=STAC_SEARCH_RE, json=_two_feature_search_response() ) @@ -282,7 +281,7 @@ def test_get_ratings_skip_warning_escalates_to_error(httpx_mock): def test_get_ratings_download_429_is_resumable_not_skipped(httpx_mock): """A rate-limited download must never be skipped -- it is raised as a resumable interruption, and resuming completes the batch. The escalation - filter proves no ``SkippedRatingWarning`` fires along the way.""" + filter proves no ``SkippedRatingWarning`` is emitted.""" httpx_mock.add_response( method="GET", url=STAC_SEARCH_RE, json=_stub_search_response() ) @@ -301,8 +300,8 @@ def test_get_ratings_download_429_is_resumable_not_skipped(httpx_mock): def test_stac_next_link_refuses_another_host(httpx_mock): """The STAC page walk must not follow a link off the ratings host. - The search request carries the Water Data API key; a ``next`` href naming - another host would take it somewhere the caller never asked for. Unlike the + The search request includes the Water Data API key; a ``next`` href naming + another host would send it to a host the caller never asked for. Unlike the OGC engine, this walk had no host check at all. """ httpx_mock.add_response( @@ -318,7 +317,7 @@ def test_stac_next_link_refuses_another_host(httpx_mock): def test_stac_next_link_strips_embedded_credentials(httpx_mock): - """A same-host ``next`` href must not smuggle in ``user:pass@``. + """A same-host ``next`` href must not include ``user:pass@``. The host check passes by construction here, so only the strip catches it. """ diff --git a/tests/waterdata_test.py b/tests/waterdata_test.py index 99938de48..80154b47a 100644 --- a/tests/waterdata_test.py +++ b/tests/waterdata_test.py @@ -54,7 +54,7 @@ #: Two real features per collection, captured from the live collection and trimmed. #: Property names, nesting, and value types (including the numeric-looking -#: strings the API really sends) are verbatim; only the row count is reduced. +#: strings the API sends) are verbatim; only the row count is reduced. #: Regenerate a collection by re-querying it with ``limit=2`` and replacing that #: key -- the getters' behavior depends on the shape, not the row count. _OGC_FIXTURES = json.loads( @@ -272,8 +272,8 @@ def spy(**kwargs): def test_legacy_camelcase_kwargs_return_identical_to_snake_case(httpx_mock): """End-to-end: a legacy camelCase ``get_samples`` call returns results byte-identical to the equivalent snake_case call — same request URL and same - DataFrame — for every renamed parameter at once. The camelCase shim changes - nothing the caller sees but the parameter names.""" + DataFrame — for every renamed parameter at once. The camelCase shim changes nothing + for the caller but the parameter names.""" import warnings from dataretrieval.waterdata.api import _SAMPLES_LEGACY_KWARGS @@ -356,7 +356,7 @@ def test_construct_api_requests_monitoring_locations_post(): assert req.method == "POST" assert req.headers["Content-Type"] == "application/query-cql-json" - # Body is serialized compactly (tight separators, no whitespace): the + # Body is serialized compactly (compact separators, no whitespace): the # body counts against the server's ~8 KB request-size cap and the # chunk planner's byte budget, so pretty-printing would needlessly # halve how many ids fit per chunk and double the chunk count. @@ -377,7 +377,7 @@ def test_construct_api_requests_monitoring_locations_post(): def test_construct_cql_request_post_verbatim_body(): """get_cql's request builder POSTs the CQL2 body verbatim with the - right content-type, and puts the OGC knobs on the URL.""" + right content-type, and puts the OGC query parameters on the URL.""" body = json.dumps( {"op": "like", "args": [{"property": "hydrologic_unit_code"}, "02070010%"]}, separators=(",", ":"), @@ -406,7 +406,8 @@ def test_construct_cql_request_post_verbatim_body(): def test_construct_cql_request_skip_geometry_none_omits_param(): """skip_geometry=None leaves skipGeometry unset (server default), so it never - reaches the URL — matching get_cql's default.""" + appears in the URL — matching get_cql's default. + """ req = _construct_cql_request("daily", "{}") assert "skipGeometry" not in str(req.url) @@ -416,7 +417,7 @@ def test_get_cql_service_keyword_is_deprecated_but_works(): ``service`` was the published spelling, and OGC API - Features calls the value a collection -- it is the ``collectionId`` in ``/collections/{id}``. - The rename must not silently change behavior for callers using the old name. + The rename must not change behavior for callers using the old name. """ with pytest.warns(DeprecationWarning, match="use 'collection'"): with pytest.raises(ValueError, match="Invalid collection"): @@ -428,8 +429,8 @@ def test_get_cql_service_keyword_is_deprecated_but_works(): with pytest.raises(ValueError, match="Invalid collection"): get_cql(collection="not-a-collection", cql="a=1") - # Passing both spellings is ambiguous and refused, which the hand-rolled - # shim this replaced did not do -- it silently dropped ``service``. + # Passing both spellings is ambiguous and refused, which the hand-written shim this + # replaced did not do -- it dropped ``service`` without a warning. with pytest.raises(TypeError, match="received both"): get_cql(service="daily", collection="daily", cql="a=1") @@ -454,7 +455,7 @@ def test_waterdata_services_literal_matches_output_id_map(): def test_construct_api_requests_single_value_stays_get(): - """A length-1 list (or scalar) reaches the URL as a plain value, not a + """A length-1 list (or scalar) appears in the URL as a plain value, not a comma-separated form, so existing single-site callers see no change.""" req = _construct_api_requests( "daily", @@ -468,8 +469,8 @@ def test_construct_api_requests_single_value_stays_get(): def test_construct_api_requests_numeric_list_joins_with_str(): """Numeric-list params (e.g. ``water_year=[2020, 2021]`` on get_peaks) - must reach the URL as a comma-joined string, not crash on ``",".join`` - of ints. The generator-of-``str(x)`` exists exactly for this case.""" + must appear in the URL as a comma-joined string, not fail on ``",".join`` + of ints. The generator-of-``str(x)`` exists for this case.""" req = _construct_api_requests( "peaks", monitoring_location_id="USGS-05427718", @@ -504,7 +505,7 @@ def test_construct_api_requests_two_element_date_list_becomes_interval(): """A two-element date list is interpreted as start/end of an OGC datetime interval (joined with '/'), NOT as two discrete dates. The OGC `datetime` parameter does not support "these N specific dates" — that would require - a CQL filter. Verifying so this contract is locked in.""" + a CQL filter. Verifying so this contract is pinned.""" req = _construct_api_requests( "daily", monitoring_location_id="USGS-05427718", @@ -519,15 +520,14 @@ def test_construct_api_requests_two_element_date_list_becomes_interval(): # These replace what used to be ~34 live calls to the Water Data API. Each one # serves a committed fixture (``tests/data/waterdata_ogc_fixtures.json``, two # real features per collection captured from the collection) and asserts what we -# actually control: that the request we build carries the right params, and that -# the frame we hand back has the right columns, dtypes, and ordering. +# control: that the request this package builds has the right params, and that +# the frame returned has the right columns, dtypes, and ordering. # -# The assertions they replaced could not do that. ``len(df) > 0`` passes or fails -# on whether a particular gage reported yesterday; ``df.shape[1] == 97`` breaks -# when USGS adds a column, which is not our bug. Genuine upstream-drift -# detection lives in ``waterdata_queryables_test.py`` (marked ``live``), which -# diffs each collection's queryables against a snapshot and tells us precisely -# what moved. +# The assertions they replaced could not do that. ``len(df) > 0`` passes or fails on +# whether a particular gage reported yesterday; ``df.shape[1] == 97`` breaks when USGS +# adds a column, which is not a defect in this package. Upstream-drift detection is in +# ``waterdata_queryables_test.py`` (marked ``live``), which diffs each collection's +# queryables against a snapshot and tells us what changed. def _fixture(collection): @@ -642,7 +642,7 @@ def test_samples_service_profile_routes_to_its_endpoint( Previously one live test per collection asserted a column count against real data (``len(df.columns) == 97``), which broke whenever the collection added a - field. What is ours to get right is the routing and the parse, so that is + field. What this package controls is the routing and the parse, so that is what this checks. """ httpx_mock.add_response( @@ -662,7 +662,7 @@ def test_samples_service_profile_routes_to_its_endpoint( def test_get_daily(httpx_mock): - """A daily query returns tidy rows with the collection id renamed to + """A daily query returns rows with the collection id renamed to ``daily_id`` and moved last, dates as ``date`` objects, values numeric.""" _mock_items(httpx_mock, "daily") @@ -719,9 +719,10 @@ def test_get_daily_properties(httpx_mock): assert df.columns[0] == "daily_id" assert df.columns[-1] == "geometry" assert df.shape[1] == len(requested) - # ``daily_id`` is our name for the wire's ``id`` and ``geometry`` is governed - # by ``skipGeometry``, not by ``properties`` -- neither is a real queryable, - # so neither may be forwarded or the collection would reject the projection. + # ``daily_id`` is this package's name for the wire ``id`` and ``geometry`` is + # governed by ``skipGeometry``, not by ``properties`` -- neither is a real + # queryable, so neither may be forwarded or the collection would reject the + # projection. sent = _sent(httpx_mock, "daily")[0]["properties"][0].split(",") assert "daily_id" not in sent and "geometry" not in sent assert sent == ["monitoring_location_id", "parameter_code", "time", "value"] @@ -763,7 +764,7 @@ def test_get_daily_no_geometry(httpx_mock): def test_get_daily_empty_no_geometry(httpx_mock): - """An empty skip-geometry request has the same plain shape as a hit.""" + """An empty skip-geometry request has the same plain shape as a non-empty result.""" httpx_mock.add_response( method="GET", url=_schema_url("daily"), @@ -834,7 +835,7 @@ def test_get_latest_daily(httpx_mock): def test_get_latest_daily_properties_geometry(httpx_mock): - """Geometry survives an explicit ``properties`` list that omits it -- the + """Geometry is kept through an explicit ``properties`` list that omits it -- the collection returns it regardless unless ``skip_geometry`` is set, so the projection must not drop it.""" _mock_items(httpx_mock, "latest-daily") @@ -966,7 +967,7 @@ def test_get_cql_like_wildcard(httpx_mock): def test_get_cql_resume_returns_finalized_shape(httpx_mock): """A resumed ``get_cql`` returns the same finished ``(df, BaseMetadata)`` - shape as an uninterrupted call. The verbatim-CQL path drives the shared + shape as an uninterrupted call. The verbatim-CQL path runs through the shared executor with the same finalizer as the typed getters, so ``exc.call.resume()`` yields the shaped result, not a raw ``(frame, response)`` pair.""" @@ -1027,8 +1028,7 @@ def test_get_field_measurements_metadata(httpx_mock): def test_get_field_measurements_metadata_multi_site(httpx_mock): - """Multiple sites plus a parameter filter reach the collection in one - request.""" + """Multiple sites and a parameter filter are sent in one request.""" sites = ["USGS-07069000", "USGS-07064000", "USGS-07068000"] _mock_items(httpx_mock, "field-measurements-metadata") @@ -1137,7 +1137,7 @@ def test_get_peaks_water_year_filter(httpx_mock): The live version of this test asserted only that the returned rows fell inside the requested years -- which an empty frame satisfies, so it could - not fail. Asserting on the outgoing request is what actually pins the + not fail. Asserting on the outgoing request is what pins the behavior. """ _mock_items(httpx_mock, "peaks") @@ -1180,7 +1180,7 @@ def test_get_reference_table(httpx_mock): def test_get_reference_table_rejects_unknown_collection_by_its_own_name(httpx_mock): - """The rejection names ``collection`` -- the parameter actually passed. + """The rejection names ``collection`` -- the parameter passed. Regression: this check was copied from ``get_codes``, message and local variable name included, so an unknown ``collection=`` was reported as an @@ -1195,10 +1195,10 @@ def test_get_reference_table_serves_countries(httpx_mock): """``countries`` is a real reference collection and singularizes to ``country``. - It sits beside ``counties`` in the service catalog but was missing from the - accepted vocabulary, so the rejection told a caller asking for a real - collection that it did not exist. The shared ``-s`` rule would also have - named its id column ``countrie``. + It is listed with ``counties`` in the service catalog but was missing from the + accepted vocabulary, so the rejection told a caller requesting a real collection + that it did not exist. The shared ``-s`` rule would also have named its id column + ``countrie``. """ _mock_items( httpx_mock, @@ -1234,14 +1234,13 @@ def test_get_reference_table_with_query(httpx_mock): def test_get_daily_max_rows_is_excluded_from_request_and_forwarded(): - # ``max_rows`` is a client-side pagination cap, not an OGC query - # parameter — the server never sees it. So a getter must keep it out of - # the request ``args`` (which become query params) and instead forward it - # to ``get_ogc_data`` as the keyword that drives the cap. This pins that - # wiring; the cap mechanism itself (stop following ``next`` once the cap is - # met, then truncate the combined frame to exactly N) is covered without a - # network round-trip by the ``_row_cap`` / ``_finalize_ogc`` tests in - # tests/waterdata_utils_test.py. + # ``max_rows`` is a client-side pagination cap, not an OGC query parameter — the + # server never sees it. So a getter must keep it out of the request ``args`` (which + # become query params) and instead forward it to ``get_ogc_data`` as the keyword + # that sets the cap. This pins that connection; the cap mechanism itself (stop + # following ``next`` once the cap is met, then truncate the combined frame to + # exactly N) is covered without a network round-trip by the ``_row_cap`` / + # ``_finalize_ogc`` tests in tests/waterdata_utils_test.py. with mock.patch("dataretrieval.waterdata.time_series.get_ogc_data") as fake: fake.return_value = (pd.DataFrame(), mock.MagicMock(spec=[])) get_daily( @@ -1260,7 +1259,7 @@ def test_get_cql_max_rows_is_excluded_from_request_and_forwarded(): It was the only one without ``max_rows``, and its ``limit`` is the page size -- so asking for a few rows through ``limit`` instead paged the whole match a few rows at a time. A bounded probe written that way spent ~400 - requests of an hourly quota of 1000 before the service refused it. + requests of an hourly quota of 1000 before the service rejected it. """ with mock.patch("dataretrieval.waterdata.cql.get_ogc_data") as fake: fake.return_value = (pd.DataFrame(), mock.MagicMock(spec=[])) @@ -1281,9 +1280,9 @@ def test_get_reference_table_wrong_name(): @pytest.mark.parametrize("bad", [0, -1, 2.5, 10.0, True]) def test_get_reference_table_rejects_bad_max_rows(bad): - # max_rows must be a genuine positive int; a non-positive value, a float + # max_rows must be a positive int; a non-positive value, a float # (even integral like 10.0), or a bool must raise ValueError up front — - # not crash later inside pandas .head(). Raises before any HTTP request. + # not raise later inside pandas .head(). Raises before any HTTP request. with pytest.raises(ValueError, match="positive integer"): get_reference_table("agency-codes", max_rows=bad) @@ -1301,7 +1300,7 @@ def test_get_reference_table_accepts_numpy_int_max_rows(httpx_mock): # --- statistics -------------------------------------------------------------- # The statistics API nests its values two levels deep (feature -> data -> -# values); these pin the flattening, which is the part we own. +# values); these pin the flattening, which is the part this package controls. def _mock_stats(httpx_mock, collection): @@ -1346,7 +1345,7 @@ def test_get_stats_por(httpx_mock): def test_get_stats_por_expanded_false(httpx_mock): """``expand_percentiles=False`` keeps the raw ``percentiles`` list column - instead of exploding it into one row per percentile.""" + instead of expanding it into one row per percentile.""" _mock_stats(httpx_mock, "observationNormals") df, _ = get_stats_por( @@ -1370,7 +1369,7 @@ def test_get_stats_por_expanded_false(httpx_mock): def test_get_stats_date_range(httpx_mock): - """Interval statistics carry an ``interval_type`` distinguishing month from + """Interval statistics have an ``interval_type`` distinguishing month from calendar- and water-year rows.""" _mock_stats(httpx_mock, "observationIntervals") @@ -1403,11 +1402,11 @@ def test_valid_string(self): assert _check_monitoring_location_id("USGS-01646500") == "USGS-01646500" def test_integer_raises_type_error(self): - """An integer ID raises TypeError with a helpful AGENCY-ID hint.""" + """An integer ID raises TypeError with an AGENCY-ID format hint.""" with pytest.raises(TypeError, match="not int") as exc_info: _check_monitoring_location_id(5129115) - # The wrapper appends the AGENCY-ID format hint that the generic - # helper alone doesn't carry. + # The wrapper appends the AGENCY-ID format hint that the generic helper alone + # does not include. assert "USGS-01646500" in str(exc_info.value) def test_missing_agency_prefix_raises_value_error(self): @@ -1433,7 +1432,7 @@ def test_get_daily_malformed_id_raises(self): def test_per_item_format_check_in_list(self): """The AGENCY-ID format check runs on EVERY element of an iterable, not just the first. Regression guard against a - future ``_check_monitoring_location_id`` loop that bails after one + future ``_check_monitoring_location_id`` loop that stops after one valid item or only checks the head.""" with pytest.raises(ValueError, match="Invalid monitoring_location_id"): _check_monitoring_location_id(["USGS-01646500", "badformat"]) @@ -1442,7 +1441,7 @@ def test_per_item_format_check_in_list(self): class TestNormalizeStrIterable: """Tests for the generic _normalize_str_iterable helper. - Mirrors TestCheckMonitoringLocationId for the type/iterable contract; + Matches TestCheckMonitoringLocationId for the type/iterable contract; the AGENCY-ID format check is monitoring_location_id-specific and lives only in the _check_monitoring_location_id wrapper. """ @@ -1485,7 +1484,7 @@ def test_dict_raises_type_error(self): _normalize_str_iterable({"00060": "discharge"}, "parameter_code") def test_get_daily_parameter_code_as_series(self): - """Wiring check: pd.Series for ``parameter_code`` arrives at the inner + """Integration check: pd.Series for ``parameter_code`` is passed to the inner call as a list. Regression for the gap PR #229 originally left on every multi-value @@ -1507,12 +1506,12 @@ def test_get_daily_parameter_code_as_series(self): assert isinstance(args_dict["parameter_code"], list) def test_list_of_ints_rejected_at_boundary(self): - """List-of-non-strings must be caught client-side, not silently sent. + """List-of-non-strings must be caught client-side, not sent unchecked. - Regression: an earlier pass through ``_get_args`` had a - ``list-of-non-str`` fast-path that bypassed normalization, so - ``parameter_code=[60, 65]`` would reach the OGC API and surface as - a confusing JSONDecodeError on the malformed response. + Regression: an earlier pass through ``_get_args`` had a ``list-of-non-str`` + fast-path that bypassed normalization, so ``parameter_code=[60, 65]`` would be + sent to the OGC API and be raised as an unclear JSONDecodeError on the malformed + response. """ with pytest.raises(TypeError, match="parameter_code elements must be strings"): get_daily( @@ -1531,10 +1530,9 @@ def test_get_reference_table_forwards_limit_as_a_query_arg(): def test_get_reference_table_docstring_lists_every_collection(): - """The docstring enumerates the vocabulary by hand, so it drifts the - moment a collection is added -- ``countries`` was served, accepted, and - absent from the docs. A reader who trusts the prose must not be told a - real collection does not exist. + """The docstring enumerates the vocabulary by hand, so it drifts as soon as a + collection is added -- ``countries`` was served, accepted, and absent from the docs. + A reader who trusts the prose must not be told a real collection does not exist. """ from typing import get_args diff --git a/tests/waterdata_utils_test.py b/tests/waterdata_utils_test.py index 2f4c45bcd..f0d6eaf2c 100644 --- a/tests/waterdata_utils_test.py +++ b/tests/waterdata_utils_test.py @@ -63,12 +63,12 @@ def _run_walk_pages(*, geopd, req, client, row_cap=None): - """Drive the async ``_walk_pages`` to completion synchronously. + """Run the async ``_walk_pages`` to completion synchronously. The chunker core is async-only now, so these tests build an ``AsyncMock(spec=httpx.AsyncClient)`` whose ``.send``/``.request`` are awaitable and run the coroutine via ``asyncio.run``. This thin shim - keeps the historical sync-shaped call sites terse while exercising the + keeps the older synchronous call sites short while exercising the real async pagination loop. """ return asyncio.run( @@ -107,7 +107,7 @@ def test_get_args_empty(): def test_walk_pages_multiple_mocked(): # Setup mock responses resp1 = mock.MagicMock() - # A real response always carries an ``httpx.URL``; the next-page check + # A real response always has an ``httpx.URL``; the next-page check # resolves and host-checks the ``next`` link against it. resp1.url = httpx.URL("https://example.com/page1") resp1.json.return_value = { @@ -199,7 +199,7 @@ def _page(idx, *, has_next): mock_client = mock.AsyncMock(spec=httpx.AsyncClient) mock_client.send.return_value = _page(1, has_next=True) - # page 2 still advertises a ``next`` (page 3) that must never be fetched. + # page 2 still includes a ``next`` (page 3) that must never be fetched. mock_client.request.return_value = _page(2, has_next=True) mock_req = mock.MagicMock(spec=httpx.Request) @@ -216,7 +216,7 @@ def _page(idx, *, has_next): def test_finalize_ogc_truncates_combined_to_max_rows(): # max_rows is enforced on the *combined* frame in _finalize_ogc (after # dedup/sort), so it bounds the total exactly even when a chunked call's - # per-chunk pages overshoot the per-_paginate early-stop. + # per-chunk pages exceed the per-_paginate early-stop. frame = pd.DataFrame({"id": [str(i) for i in range(10)]}) resp = mock.MagicMock() resp.url = "https://example.com/q" @@ -237,7 +237,7 @@ def test_finalize_ogc_truncates_combined_to_max_rows(): def _resp_ok(features): - """Build a 200-OK mock response carrying the given features list.""" + """Build a 200-OK mock response with the given features list.""" links = [{"rel": "next", "href": "https://example.com/page2"}] if features else [] resp = mock.MagicMock() resp.json.return_value = { @@ -293,15 +293,15 @@ def test_walk_pages_raises_with_class_name_when_cause_stringifies_empty(): msg = str(excinfo.value) assert "Timeout" in msg, msg - # Sanity-check the malformed-empty placeholder didn't slip through. + # Sanity-check the malformed-empty placeholder was not emitted. assert "page(s): ." not in msg assert "page(s): To recover" not in msg def test_walk_pages_raises_on_5xx_mid_pagination(): """A 5xx mid-pagination must raise — partial data is no longer returned - because the API has no resume cursor, so silently truncating would return - an incomplete frame the caller cannot tell from a complete one.""" + because the API has no resume cursor, so truncating without an error would return + an incomplete frame the caller cannot distinguish from a complete one.""" page2_503 = mock.MagicMock() page2_503.status_code = 503 page2_503.json.return_value = { @@ -334,12 +334,11 @@ def test_walk_pages_raises_on_mid_pagination_429(): def test_walk_pages_wraps_initial_page_parse_error(): - """A 200 response whose body fails to parse on the FIRST page used - to escape ``_walk_pages`` as a raw ``JSONDecodeError``, while the - SAME failure on a subsequent page was wrapped via - ``_paginated_failure_message``. The asymmetry meant operators got - different exception types for the same logical bug depending on - which page hit it. The initial-parse wrapper closes the gap.""" + """A 200 response whose body fails to parse on the FIRST page used to propagate out + of ``_walk_pages`` as a raw ``JSONDecodeError``, while the SAME failure on a + subsequent page was wrapped via ``_paginated_failure_message``. The asymmetry meant + operators got different exception types for the same logical bug depending on + which page encountered it. The initial-parse wrapper removes the asymmetry.""" resp = mock.MagicMock() resp.status_code = 200 resp.url = "https://example.com/page1" @@ -357,17 +356,16 @@ def test_walk_pages_wraps_initial_page_parse_error(): with pytest.raises(DataRetrievalError, match="Paginated request failed") as excinfo: _run_walk_pages(geopd=False, req=mock_req, client=mock_client) - # The JSONDecodeError causing it is on __cause__ so callers can drill in. + # The JSONDecodeError causing it is on __cause__ so callers can inspect it. assert isinstance(excinfo.value.__cause__, json.JSONDecodeError) def test_get_resp_data_handles_missing_features_key(): - """Regression: a 200 with ``numberReturned > 0`` but no - ``features`` key (real schema-drift shape) used to crash - ``_get_resp_data`` with ``KeyError`` — wrapped downstream by - ``_paginate`` as a generic transport error. ``_handle_nesting`` - was already hardened against this; ``_get_resp_data`` now mirrors - that defensiveness and returns an empty frame instead.""" + """Regression: a 200 with ``numberReturned > 0`` but no ``features`` key (a real + schema-drift shape) used to fail ``_get_resp_data`` with ``KeyError`` — wrapped + downstream by ``_paginate`` as a generic transport error. ``_handle_nesting`` + already handled this; ``_get_resp_data`` now does the same and returns an empty + frame instead.""" resp = mock.Mock() resp.json.return_value = {"numberReturned": 1, "links": []} df = _get_resp_data(resp, geopd=False) @@ -377,11 +375,12 @@ def test_get_resp_data_handles_missing_features_key(): def test_next_req_url_follows_link_without_number_returned(): """The NGWMN OGC API omits ``numberReturned`` from its page envelope, so - ``_next_req_url`` keys the ``next`` link off ``features`` (mirroring - ``_get_resp_data``) rather than that count -- otherwise a page that carries - features but no count stops pagination after page 1 and silently truncates - every multi-page result. A page that carries features still follows its - ``next`` link even when ``numberReturned`` is absent.""" + ``_next_req_url`` keys the ``next`` link off ``features`` (matching + ``_get_resp_data``) rather than that count -- otherwise a page that has + features but no count stops pagination after page 1 and truncates every + multi-page result without an error. A page that has features still follows + its ``next`` link even when ``numberReturned`` is absent. + """ resp = mock.MagicMock() resp.url = httpx.URL("https://example.com/page1") body = { @@ -393,8 +392,9 @@ def test_next_req_url_follows_link_without_number_returned(): def test_next_req_url_stops_when_no_features(): - """A page with no features ends pagination regardless of any stray - ``next`` link (and regardless of ``numberReturned``).""" + """A page with no features ends pagination regardless of any ``next`` link present + (and regardless of ``numberReturned``). + """ resp = mock.MagicMock() resp.url = httpx.URL("https://example.com/page1") body = {"features": [], "links": [{"rel": "next", "href": "https://x/2"}]} @@ -403,7 +403,7 @@ def test_next_req_url_stops_when_no_features(): def test_walk_pages_does_not_mutate_initial_response(): """The aggregated response returned from ``_walk_pages`` is built - via ``_merge_response``, which returns a fresh copy. + via ``_merge_response``, which returns a new copy. Any caller that inspected ``initial_response.headers`` / ``.elapsed`` before pagination completed (a Session response hook, a logging middleware) must continue to see the original first-page @@ -450,7 +450,7 @@ def test_walk_pages_does_not_mutate_initial_response(): assert page1.headers["x-ratelimit-remaining"] == "999" assert page1.elapsed == page1_initial_elapsed - # The returned aggregate carries page-2 headers + cumulative elapsed. + # The returned aggregate has page-2 headers + cumulative elapsed. assert final.headers["x-ratelimit-remaining"] == "998" assert final.elapsed == datetime.timedelta(seconds=3) # And mutating the aggregate's headers doesn't leak into either page. @@ -474,7 +474,7 @@ def _stats_initial_ok(): def _run_get_data_with_failure(failure_resp_or_exc, monkeypatch): """Exercise get_data where the initial response succeeds and the - paginated follow-up fails as given. Mirrors _walk_pages_with_failure. + paginated follow-up fails as given. Matches _walk_pages_with_failure. `monkeypatch` stubs ``_handle_nesting`` so the synthetic minimal response body doesn't need to parse — these tests only assert on the pagination loop's error surfacing.""" @@ -506,7 +506,7 @@ def test_get_data_raises_on_mid_pagination_failure(monkeypatch): ``get_data`` mid-pagination case proves the stats-specific follow-up callback is wired into ``_paginate``. - Statistics drives that page walk as a one-item ``FanOut``, the same + Statistics runs that page walk as a one-item ``FanOut``, the same executor every other getter uses, so a transient mid-walk failure is resumable here too rather than ending the call outright. """ @@ -522,7 +522,7 @@ def test_get_data_raises_on_mid_pagination_failure(monkeypatch): assert "Paginated request failed" in str(paginated) assert isinstance(paginated.__cause__, httpx.ConnectError) assert "stats-boom" in str(paginated) - # Nothing completed, so there is nothing to hand back but the handle. + # Nothing completed, so nothing is returned but the handle. assert excinfo.value.call.completed_chunks == 0 @@ -580,7 +580,7 @@ def test_handle_nesting_tolerates_missing_drop_columns(): def test_handle_nesting_returns_empty_on_empty_features(): """A mid-pagination empty page ({\"features\": [], \"next\": }) - must not crash the downstream merge with + must not fail the downstream merge with ``KeyError: 'monitoring_location_id'``. The function short- circuits to an empty DataFrame so pagination can continue.""" df = _handle_nesting({"features": [], "next": None}, geopd=False) @@ -592,8 +592,8 @@ def test_handle_nesting_empty_preserves_geopd_type(): must return a ``GeoDataFrame`` rather than a plain ``DataFrame``. Otherwise a subsequent ``pd.concat([empty, geo_page])`` downgrades the final result to a plain ``DataFrame`` and strips geometry/CRS - — a real regression for geopd-installed users on stats queries - that hit an empty intermediate page.""" + — a regression for geopd-installed users on stats queries + that encountered an empty intermediate page.""" # Monkeypatch a stub gpd so the test runs whether or not geopandas is # installed. The empty-page short-circuit delegates to the shared # ``shaping._empty_feature_frame``, which resolves ``gpd`` from the shaping @@ -693,7 +693,7 @@ class _Sentinel: @pytest.mark.skipif(not _shaping_module.GEOPANDAS, reason="requires geopandas") def test_get_resp_data_keeps_one_geospatial_type_when_geometry_is_missing(): - """A spatial request cannot change frame family based on page contents.""" + """A spatial request cannot change frame type based on page contents.""" import geopandas as gpd empty = _get_resp_data(_resp_ok([]), geopd=True) @@ -713,9 +713,9 @@ def test_get_resp_data_keeps_one_geospatial_type_when_geometry_is_missing(): def test_get_resp_data_attaches_wgs84_crs(): - """A geometry-bearing Water Data page should come back tagged as - EPSG:4326 (the CRS the coordinates are published in), so callers can - run ``to_crs`` / spatial joins without first patching in a CRS by + """A geometry-bearing Water Data page should be returned with the CRS EPSG:4326 (the + CRS the coordinates are published in), so callers can run ``to_crs`` / spatial joins + without first patching in a CRS by hand. Regression for the ``.crs is None`` reported in issue #342.""" geopandas = pytest.importorskip("geopandas") @@ -735,8 +735,9 @@ def test_get_resp_data_attaches_wgs84_crs(): def test_handle_nesting_attaches_wgs84_crs(): - """The stats path builds its GeoDataFrame the same way, so it should - carry EPSG:4326 too (issue #342).""" + """The stats path builds its GeoDataFrame the same way, so it should have EPSG:4326 + too (issue #342). + """ geopandas = pytest.importorskip("geopandas") body = { @@ -765,7 +766,7 @@ def test_handle_nesting_attaches_wgs84_crs(): def test_handle_nesting_tolerates_missing_features_key(): - """A 200 response with a body that doesn't carry ``features`` at + """A 200 response with a body that does not include ``features`` at all (rare but seen in error envelopes) must also short-circuit rather than KeyError before the schema-aware extraction even runs.""" @@ -775,10 +776,9 @@ def test_handle_nesting_tolerates_missing_features_key(): def test_get_resp_data_always_materializes_id_column(): """``_get_resp_data`` must always materialize the ``id`` column - (NaN-filled when no feature carries one) so the downstream + (NaN-filled when no feature has one) so the downstream ``_arrange_cols`` rename to the collection-specific output_id - (``daily_id``, ``channel_measurements_id``, etc.) isn't a - silent no-op.""" + (``daily_id``, ``channel_measurements_id``, etc.) is not an ignored no-op.""" resp = mock.MagicMock() resp.json.return_value = { "numberReturned": 2, @@ -849,7 +849,7 @@ def test_arrange_cols_swaps_id_in_returned_columns(): def test_arrange_cols_keeps_geometry_when_present(): - """Geometry must come along even if the caller didn't list it.""" + """Geometry must be included even if the caller did not list it.""" df = pd.DataFrame({"id": ["a"], "value": [1.0], "geometry": ["p1"]}) result = _arrange_cols(df, ["value"], output_id="daily_id") assert "geometry" in result.columns @@ -918,7 +918,7 @@ def test_format_api_dates_treats_an_all_blank_sequence_as_no_filter(): def test_format_api_dates_rejects_more_than_two_values(): """A date filter is an instant, a duration, or a closed interval. Three values is a caller who meant something else, and the message says which - shapes exist rather than truncating silently.""" + shapes exist rather than truncating without an error.""" with pytest.raises(ValueError) as excinfo: _format_api_dates(["2024-01-01", "2024-06-01", "2024-12-31"], name="time") message = str(excinfo.value) @@ -927,9 +927,8 @@ def test_format_api_dates_rejects_more_than_two_values(): def test_the_duration_example_is_withheld_where_durations_are_rejected(): - """``get_ratings`` refuses ISO 8601 durations, so the shared message must - not offer 'P7D' on that path -- a caller following it would be sent - straight into a second rejection.""" + """``get_ratings`` refuses ISO 8601 durations, so the shared message must not offer + 'P7D' on that path -- a caller following it would receive a second rejection.""" with pytest.raises(ValueError) as allowed: _format_api_dates(["a", "b", "c"], name="time") with pytest.raises(ValueError) as refused: @@ -952,7 +951,7 @@ def test_format_api_dates_names_the_callers_parameter(): def test_format_api_dates_rejects_mapping(): - """`time={"2024-01-01": "x"}` would silently materialize as the keys list, + """`time={"2024-01-01": "x"}` would materialize as the keys list, accepting input the user clearly didn't intend. """ with pytest.raises(TypeError, match="date input must be a string or sequence"): @@ -986,7 +985,7 @@ def test_error_body_handles_non_json_html_response(): def test_error_body_handles_empty_response_body(): - """An empty error body returns a status/reason message without crashing.""" + """An empty error body returns a status/reason message without raising.""" resp = _make_response(500, "", reason="Internal Server Error") msg = _error_body(resp) assert msg == "500: Internal Server Error." @@ -1017,7 +1016,7 @@ def test_error_body_still_parses_well_formed_json(): def test_parse_retry_after_handles_none_and_empty(): """Absent or empty header → ``None`` (no quota signal). The chunker - treats ``None`` as "fall back to my own retry policy," so this + treats ``None`` as a signal to use its own retry policy, so this branch must not return a misleading 0.""" assert _parse_retry_after(None) is None assert _parse_retry_after("") is None @@ -1026,7 +1025,7 @@ def test_parse_retry_after_handles_none_and_empty(): def test_parse_retry_after_parses_delta_seconds(): """Integer and float forms of delta-seconds (the common shape USGS - sends) are parsed directly without touching the HTTP-date branch.""" + sends) are parsed directly without entering the HTTP-date branch.""" assert _parse_retry_after("120") == 120.0 assert _parse_retry_after("0") == 0.0 assert _parse_retry_after("42.5") == 42.5 @@ -1035,7 +1034,7 @@ def test_parse_retry_after_parses_delta_seconds(): def test_parse_retry_after_clamps_negative_delta_to_zero(): - """A negative delta-seconds means the server is saying "retry now." + """A negative delta-seconds means retry immediately. Returning the negative value would let callers pass it to ``time.sleep`` and get a ``ValueError`` — clamp at the source.""" assert _parse_retry_after("-10") == 0.0 @@ -1045,16 +1044,16 @@ def test_parse_retry_after_clamps_negative_delta_to_zero(): def test_parse_retry_after_supports_http_date_and_rejects_garbage(): """Both standard header forms are accepted; malformed values use backoff. - A date is converted to seconds exactly like the delta-seconds form, however - far out it lands: an over-long wait stops the retry and travels to the - caller on ``.retry_after`` rather than being silently ignored. + A date is converted to seconds the same way as the delta-seconds form, however + far in the future it is: an over-long wait stops the retry and reaches the + caller on ``.retry_after`` rather than being discarded. """ assert _parse_retry_after("not-a-date") is None assert _parse_retry_after("Wed, 21 Oct 2099 07:28:00 GMT") > 0 def test_raise_for_non_200_raises_service_unavailable_for_5xx(): - """5xx must surface as the typed ``ServiceUnavailable`` so the chunker can + """5xx must be raised as the typed ``ServiceUnavailable`` so the chunker can wrap it as a resumable ``ServiceInterrupted`` rather than treating it as a fatal error.""" resp = _make_response(503, "", reason="Service Unavailable") @@ -1065,9 +1064,9 @@ def test_raise_for_non_200_raises_service_unavailable_for_5xx(): def test_raise_for_non_200_attaches_retry_after_to_rate_limited(): - """``Retry-After`` on a 429 response must travel onto - ``RateLimited.retry_after`` so the chunker can surface it on - ``QuotaExhausted.retry_after`` for callers to honor.""" + """``Retry-After`` on a 429 response must be set on + ``RateLimited.retry_after`` so the chunker can report it on + ``QuotaExhausted.retry_after`` for callers to apply.""" resp = _make_response(429, "", reason="Too Many Requests") resp.headers["Retry-After"] = "60" with pytest.raises(RateLimited) as excinfo: @@ -1076,11 +1075,11 @@ def test_raise_for_non_200_attaches_retry_after_to_rate_limited(): def test_403_reports_the_services_own_reason(): - """A 403 envelope must reach the user rather than a canned guess. + """A 403 envelope must be shown to the user rather than fixed text. - The message was fixed text naming only "query exceeding server limits", - and never read the body -- so a revoked ``API_USGS_PAT``, the most common - real 403, was reported as a query-size problem. + The message was fixed text naming only "query exceeding server limits", and never + read the body -- so a revoked ``API_USGS_PAT``, the most common 403, was reported as + a query-size problem. """ resp = _make_response( 403, @@ -1153,8 +1152,7 @@ def test_403_without_an_envelope_names_the_credential_cause(): def test_error_messages_name_the_url(): """Without the URL a failed chunk in a fan-out cannot be traced back to - the request that produced it -- the message is all the interruption - carries.""" + the request that produced it -- the message is all the interruption holds.""" request = httpx.Request("GET", "https://api.waterdata.usgs.gov/ogcapi/v0/x") resp = httpx.Response(400, content=b"", request=request) with pytest.raises(HTTPError) as excinfo: @@ -1181,7 +1179,7 @@ def test_raise_for_non_200_400_raises_http_error(): with pytest.raises(HTTPError) as excinfo: _raise_for_non_200(resp) assert excinfo.value.status_code == 400 - # Fatal, not transient: the chunker keys off ``isinstance(_, TransientError)`` + # Fatal, not transient: the chunker uses ``isinstance(_, TransientError)`` # to decide whether to wrap a failure as a resumable ChunkInterrupted. assert not isinstance(excinfo.value, TransientError) @@ -1189,7 +1187,7 @@ def test_raise_for_non_200_400_raises_http_error(): def test_next_req_url_rejects_cross_host(): """``_next_req_url`` must refuse to follow a next-page link to a different host. The original request's headers (including any - auth-like artifacts) were minted for the original host; following + auth-like artifacts) were built for the original host; following a server-supplied cross-host URL would leak them — and the URL itself could be sensitive.""" resp = mock.MagicMock() @@ -1204,11 +1202,11 @@ def test_next_req_url_rejects_cross_host(): def test_next_req_url_strips_embedded_credentials(): - """A same-host next link carrying ``user:pass@`` must not survive. + """A same-host next link with ``user:pass@`` must not be kept. The cross-host guard passes here by construction -- the host matches -- so nothing else would catch it. httpx derives ``Authorization: Basic`` from - userinfo, so following the link verbatim would mint a credential the caller + userinfo, so following the link verbatim would create a credential the caller never configured and send it alongside the real API key. """ resp = mock.MagicMock() @@ -1233,7 +1231,7 @@ def test_next_req_url_strips_embedded_credentials(): def test_check_ogc_requests_raises_typed_on_5xx(httpx_mock): """``_check_ogc_requests`` routes a non-200 through ``_raise_for_non_200``, - so a 5xx surfaces as the typed ``ServiceUnavailable`` — the same typed + so a 5xx is raised as the typed ``ServiceUnavailable`` — the same typed contract as the main data path, not a raw ``httpx`` error.""" httpx_mock.add_response( method="GET", @@ -1255,7 +1253,7 @@ def test_check_ogc_requests_raises_typed_on_5xx(httpx_mock): ("someField", "some_field"), # simple camelCase ("PascalCase", "pascal_case"), # leading capital # Runs of capitals are best-effort: only the lower->Upper boundary - # before the run is split, so the acronym stays glued to the next word. + # before the run is split, so the acronym stays joined to the next word. ("someXMLField", "some_xmlfield"), ], ) @@ -1297,7 +1295,7 @@ def fake_get_data(args, service, expand_percentiles, client=None): def test_with_state_routes_into_native_queryable(): """``_with_state`` resolves the canonical ``state`` argument into the endpoint's native queryable (any encoding -> the requested representation) - and leaves args without ``state`` untouched.""" + and leaves args without ``state`` unchanged.""" assert _utils_module._with_state({"state": "WI"}, to="name", into="state_name") == { "state_name": "Wisconsin" } @@ -1331,7 +1329,7 @@ def test_with_state_conflict_via_queryables_raises(): """A native state param arriving through ``**queryables`` (i.e. not an explicit getter parameter, as with ``get_time_series_metadata``'s ``state_code``) is flattened before the mutual-exclusion check, so combining - it with ``state`` still raises rather than silently sending both filters.""" + it with ``state`` still raises rather than sending both filters.""" with pytest.raises(ValueError, match="cannot be combined"): _utils_module._with_state( {"state": "WI", "queryables": {"state_code": "55"}}, @@ -1360,7 +1358,7 @@ def fake_get_ogc_data(args, collection, *a, **k): def test_get_ogc_data_wrapper_does_not_touch_state(): """``get_ogc_data`` no longer rewrites a ``state`` key, so a passthrough - query dict (e.g. from ``get_reference_table``) is forwarded untouched.""" + query dict (e.g. from ``get_reference_table``) is forwarded unchanged.""" captured: dict = {} def fake_engine_get_ogc_data(args, collection, output_id, **k): @@ -1380,7 +1378,7 @@ def fake_engine_get_ogc_data(args, collection, output_id, **k): def test_credential_shaped_queryables_are_rejected(name): """The denylist matches spellings, not just a few exact names. - ``x_api_key`` is the tempting one -- it mirrors the ``X-Api-Key`` header + ``x_api_key`` is the likely one -- it matches the ``X-Api-Key`` header the README documents -- and an exact-match list let it through into the query string. """ @@ -1398,13 +1396,14 @@ def test_real_queryables_still_pass_through(name): class TestWireIdSwitch: """The API keys every collection on ``id``; callers spell it after the collection (``monitoring_location_id``). The switch happens here, and - dropping an alias without carrying its value to ``id`` silently sends an + dropping an alias without moving its value to ``id`` sends an unfiltered query.""" def test_the_collection_scoped_spelling_becomes_id(self): from dataretrieval.ogc.requests import _switch_arg_id - # The collection-derived spelling wins over the getter's own id_name. + # The collection-derived spelling takes precedence over the getter's own + # id_name. out = _switch_arg_id( {"monitoring_locations_id": "USGS-01646500"}, "some_other_id", @@ -1431,7 +1430,7 @@ def test_an_explicit_id_wins_and_the_aliases_are_dropped(self): def test_extract_features_returns_none_for_a_missing_body(): """``None`` means "give the caller an empty frame". An empty features list - is a real mid-pagination shape, and letting it through would crash the + is a real mid-pagination shape, and letting it through would fail the downstream merge with a missing join key rather than returning nothing.""" from dataretrieval.waterdata.stats import _extract_features @@ -1458,8 +1457,9 @@ def test_next_req_url_parses_the_body_when_not_handed_one(): def test_next_req_url_stops_on_a_page_with_no_features(): - """A ``next`` link on a featureless page is the service's pagination - running past the end; following it would loop.""" + """A ``next`` link on a featureless page is the service's pagination continuing past + the end; following it would loop. + """ from dataretrieval.ogc.engine import _next_req_url payload = { @@ -1474,7 +1474,7 @@ def test_next_req_url_stops_on_a_page_with_no_features(): class TestOgcJsonErrorDetail: - """The service's own wording is surfaced when it sends one; anything else + """The service's own wording is used when it sends one; anything else must fall back to the status-derived message rather than raising while building an error.""" @@ -1485,7 +1485,7 @@ def test_a_non_json_body_yields_no_detail(self): assert _json_error_detail(resp) is None def test_a_json_scalar_body_yields_no_detail(self): - """A bare string or list is valid JSON but carries no error envelope.""" + """A bare string or list is valid JSON but has no error envelope.""" from dataretrieval.ogc.errors import _json_error_detail assert _json_error_detail(httpx.Response(400, json="just a string")) is None diff --git a/tests/waterservices_test.py b/tests/waterservices_test.py index 0fda62d84..05f2ec964 100644 --- a/tests/waterservices_test.py +++ b/tests/waterservices_test.py @@ -41,7 +41,7 @@ def test_query_waterdata_validation(): query_waterdata(service=None, site_no="sites") message = str(value_error.value) assert "Invalid service: None" in message - # 'ratings' was advertised here but is not an NwisWeb program: the URL it + # 'ratings' was listed here but is not an NwisWeb program: the URL it # built returned an HTML error page, not data. assert "'peaks'" in message assert "get_ratings" in message @@ -50,7 +50,7 @@ def test_query_waterdata_validation(): query_waterdata(service="pmcodes", nw_longitude_va="something") message = str(value_error.value) assert "must be given together to describe a bounding box" in message - # The three corners actually absent, so the caller knows what to add. + # The three corners absent, so the caller knows what to add. assert "nw_latitude_va, se_longitude_va and se_latitude_va" in message diff --git a/tests/wqp_test.py b/tests/wqp_test.py index 3cc6aaf8f..a2afda597 100644 --- a/tests/wqp_test.py +++ b/tests/wqp_test.py @@ -60,9 +60,9 @@ def test_get_results_opts_into_retry(monkeypatch): def test_read_wqp_csv_preserves_leading_zero_codes(): - """Regression: WQP code columns (HUCs, parameter codes, FIPS) carry - significant leading zeros; a bare ``read_csv`` inferred them as int/float - and dropped the zeros (``"00060"`` -> ``60``). ``_read_wqp_csv`` reads + """Regression: WQP code columns (HUCs, parameter codes, FIPS) have significant + leading zeros; a bare ``read_csv`` inferred them as int/float and dropped the zeros + (``"00060"`` -> ``60``). ``_read_wqp_csv`` reads code/identifier columns as ``str`` while leaving value columns numeric.""" from dataretrieval.wqp import _read_wqp_csv @@ -94,7 +94,7 @@ def test_get_results(httpx_mock): assert df.shape == (5, 65) _assert_wqp_metadata(md, request_url) assert df["ActivityStartDateTime"].notna().all() - # Regression: the getter must thread the query kwargs into the metadata + # Regression: the getter must pass the query kwargs into the metadata # (it previously built WQP_Metadata(response), dropping them), so that # md.site_info has a siteid to look up instead of always returning None. assert md._parameters.get("siteid") == "WIDNR_WQX-10032762" @@ -233,7 +233,7 @@ def test_wqp_url_profiles(builder, service, expected, warning): def test_a_configured_base_url_moves_both_interfaces(): - """One root, both paths: the portal serves legacy and WQX3 from one host. + """The portal serves legacy and WQX3 from one host, so one root covers both paths. Redirecting only the interface a caller happened to use first would leave the other pointed at the service they were redirecting away from, which is @@ -251,7 +251,7 @@ def test_a_configured_base_url_moves_both_interfaces(): assert wqx3 == f"{mirror}/wqx3/Result/search?" # Outside the block, the portal's own root again -- the redirect is scoped - # to the ``with`` statement, not latched at import. + # to the ``with`` statement, not fixed at import. with pytest.warns(DataCurrencyWarning): assert wqp.wqp_url("Result").startswith("https://www.waterqualitydata.us/") @@ -322,7 +322,7 @@ def test_wqp_url_profiles_reject_unknown_service( ids=[case[0].__name__ for case in _WHAT_CASES], ) def test_what_query(httpx_mock, func, service, fixture, profile_column): - """Each WQP ``what_*`` wrapper hits its own service endpoint and returns the + """Each WQP ``what_*`` wrapper requests its own service endpoint and returns the parsed DataFrame + metadata.""" request_url = ( f"https://www.waterqualitydata.us/data/{service}/Search?" @@ -333,7 +333,7 @@ def test_what_query(httpx_mock, func, service, fixture, profile_column): assert type(df) is DataFrame assert not df.empty assert profile_column in df.columns - # Only get_results post-processes: the shared funnel must hand back each + # Only get_results post-processes: the shared query path must return each # what_* response exactly as parsed, with no DateTime columns and no sort. with open(f"tests/data/{fixture}") as text: assert_frame_equal(df, _read_wqp_csv(text.read())) @@ -371,7 +371,7 @@ def test_credential_shaped_wqp_kwargs_are_rejected(name): "name", ["siteid", "characteristicName", "statecode", "providers", "pCode"] ) def test_real_wqp_filters_still_pass_through(name): - """The denylist must not claim names the portal owns.""" + """The denylist must not include names the portal defines.""" assert _check_kwargs({name: "v"})[name] == "v" @@ -380,7 +380,7 @@ def test_get_results_wqx3_preserves_user_dataProfile(httpx_mock): Regression: previously the `else` branch of the `dataProfile` validation triggered whenever the value was *not invalid*, including any valid - user-supplied profile, silently overwriting it with 'fullPhysChem'. + user-supplied profile, overwriting it with 'fullPhysChem'. """ request_url = ( "https://www.waterqualitydata.us/wqx3/Result/search?" @@ -419,7 +419,7 @@ def test_wqp_metadata_site_info_is_accessible_property(): def test_wqp_metadata_site_info_routes_to_what_sites(monkeypatch): - """When the query carried a ``siteid`` (WQP's site identifier), + """When the query included a ``siteid`` (WQP's site identifier), ``site_info`` delegates to ``wqp.what_sites`` with that identifier.""" import dataretrieval.wqp as wqp_mod