refactor(errors)!: unify request failures under a DataRetrievalError taxonomy#313
Merged
thodson-usgs merged 1 commit intoJun 3, 2026
Conversation
2afd2d7 to
9801177
Compare
thodson-usgs
added a commit
to thodson-usgs/dataretrieval-python
that referenced
this pull request
Jun 2, 2026
…ils/ package
## Why
`dataretrieval/waterdata/utils.py` had grown to 2033 LOC spanning ~6 unrelated
domains -- request building, response parsing, result finalization,
pagination/async, stats post-processing, and validation -- plus constants and
the public engines. It was the package's one genuine god-module. (An
architecture review found the package's OO is otherwise appropriate, so this is
a modularization, not an OO-pattern refactor.)
## What
Convert `utils.py` into a `utils/` package: the public surface stays in
`utils/__init__.py` (a thin facade) and the implementation is split across six
cohesive submodules, moving every definition verbatim (no signature/logic
changes):
| submodule | holds |
|---|---|
| `utils/constants.py` | URLs, `_OUTPUT_ID_BY_SERVICE`, regexes, param sets (dependency-free) |
| `utils/http.py` | headers, `_error_body`, `_raise_for_non_200`, retry-after |
| `utils/validate.py` | arg normalization/validation (`_get_args`, `_check_*`) |
| `utils/requests.py` | request building (`_construct_api_requests`, CQL2, dates) |
| `utils/responses.py` | geometry-agnostic parsing / finalization / stats shaping |
| `utils/engine.py` | pagination/async driver (`_paginate`, `_run_sync`, ...) |
`utils/__init__.py` re-exports the internal API (explicit `__all__`, 56 names),
so every existing `from dataretrieval.waterdata.utils import ...` and
`mock.patch("dataretrieval.waterdata.utils.<name>")` keeps working -- no import
sites or tests were touched. `dataretrieval.waterdata.utils` resolves to the
package's `__init__`, so the import path is unchanged from when it was a module.
Seven functions remain physically defined in `utils/__init__.py`
(`get_ogc_data`, `_fetch_once`, `get_stats_data`, `_get_resp_data`,
`_ogc_parse_response`, `_walk_pages`, `_handle_stats_nesting`) because the test
suite monkeypatches them (or `gpd`) by their `dataretrieval.waterdata.utils.*`
name, and a function's global lookups resolve in its defining module. The
geopandas probe stays with them, and the pagination logger keeps the name
`dataretrieval.waterdata.utils` (a caplog test pins it). These could later move
to the `engine`/`responses` submodules -- which do not import the package, so
there is no cycle -- but that requires re-targeting the test patches; left as a
follow-up.
## Behavior-preserving
- 56 top-level definitions moved verbatim -- none lost, none duplicated.
- 469 tests pass, 2 skipped; ruff clean; submodules import without cycles
(`constants` <- `http`/`validate` <- `requests`/`responses` <- `engine` <-
`__init__`); `chunking.py` untouched.
## Note
Overlaps with the error-taxonomy (DOI-USGS#313) and namespace (DOI-USGS#315) PRs on `waterdata/`
imports -- sequence on merge.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
12cd148 to
5bc68a6
Compare
…taxonomy
Before, an HTTP failure surfaced as a different exception type depending on
which module made the request -- a ValueError (or bare Exception) on the legacy
query() path, RuntimeError-based types on the waterdata path, a bare
httpx.HTTPStatusError elsewhere -- so there was no single `except` for "any
dataretrieval request failure".
Introduce dataretrieval/exceptions.py (dependency-free, re-exported at top level
as dataretrieval.<Name>), rooted at DataRetrievalError, with two intermediate
bases that name the axes a caller reasons about:
DataRetrievalError(Exception)
|- BadRequestError(.., ValueError) # 400
|- NotFoundError(.., ValueError) # 404
|- RequestTooLarge(.., ValueError) # base: request too large to satisfy
| |- URLTooLong # 414 / client-side URL reject
| '- Unchunkable # chunker planner floor
|- NoSitesError # empty result
'- TransientError(.., RuntimeError) # base: retryable; carries retry_after
|- RateLimited # 429
'- ServiceUnavailable # 5xx (both paths)
- One type per condition, raised by both the legacy query() path and the Water
Data chunker. Callers can catch a whole family (`except RequestTooLarge` /
`except TransientError`); the chunker's retry check is a single
isinstance(exc, TransientError).
- query()'s inline status ladder is extracted into a reusable _raise_for_status().
- NoSitesError now subclasses DataRetrievalError (was Exception).
- Built-in compatibility by kind: fatal client errors are also ValueError,
transient transport errors also RuntimeError, so existing `except ValueError`
/ `except RuntimeError` handlers keep working.
BREAKING CHANGES
- The legacy query() path raises typed errors instead of ad-hoc ValueErrors
(400 -> BadRequestError, 404 -> NotFoundError, 414/over-long URL -> URLTooLong).
- A 5xx on the legacy query() path now raises ServiceUnavailable, a RuntimeError
(was a ValueError): a transient server failure is a runtime condition, not a
bad value.
- The Water Data chunker's planner-floor error is Unchunkable (a RequestTooLarge
subclass).
- Import the transport types/bases from dataretrieval / dataretrieval.exceptions,
not from dataretrieval.waterdata.chunking.
Verified: 477 passed / 2 skipped, ruff clean; live API spot checks (404/400/
over-long URL raise the typed errors, 200 unaffected); all 21 example notebooks
execute end-to-end against the live API (227/227 cells, 0 errors).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5bc68a6 to
d46de2b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An HTTP failure surfaced as a different exception type depending on which module made the request, so there was no single
exceptfor "anydataretrievalrequest failure":query()(wqp,nwis,ngwmn,nldi)ValueError, orNoSitesError(Exception)waterdataRateLimited/ServiceUnavailable(RuntimeError),RequestTooLarge(ValueError),ChunkInterrupted(RuntimeError)nadp,streamstatshttpx.HTTPStatusErrorThe same HTTP condition also produced near-homonym twins —
ServiceUnavailableErrorvsServiceUnavailable,RequestTooLargeErrorvsRequestTooLarge— distinguished only by an-Errorsuffix that silently encoded which code path raised it.Change
Add
dataretrieval.exceptions(dependency-free; every type re-exported at top level asdataretrieval.<Name>), rooted atDataRetrievalError, with two intermediate bases that name the axes a caller actually reasons about:ServiceUnavailableand a too-long URL raises aRequestTooLargesubclass whether the request came from the legacyquery()path or the Water Data chunker.except RequestTooLargecovers 414 + unchunkable;except TransientErrorcovers any retryable failure; the chunker's retry check collapses to a singleisinstance(exc, TransientError).ValueError, transient transport errors alsoRuntimeError, so existingexcept ValueError/except RuntimeErrorkeep working.query()'s inline status ladder is extracted into a reusable_raise_for_status();NoSitesErrornow subclassesDataRetrievalError(wasException).Breaking changes
query()path raises typed errors instead of ad-hocValueErrors (400 →BadRequestError, 404 →NotFoundError, 414 / over-long URL →URLTooLong).query()path now raisesServiceUnavailable, aRuntimeError(previously aValueError) — a transient server failure is a runtime condition, not a bad value.Unchunkable(aRequestTooLargesubclass).dataretrieval/dataretrieval.exceptions, not fromdataretrieval.waterdata.chunking.The root catch (
except DataRetrievalError) andValueError/RuntimeErrorcompatibility (by kind) are preserved.Verification
query()path): 404 →NotFoundError, 400 →BadRequestError, an over-long URL →URLTooLong(allDataRetrievalError+ValueError); a 200 is unaffected.waterdatapath.Out of scope (proposed follow-ups)
nadp/streamstatsstill raisehttpx.HTTPStatusError— a one-line swap each to_raise_for_status, but it changes their public error type.nldi's manual non-200ValueErrorandnwis._parse_json_or_raise's HTML-detectionValueErroraren't rooted yet.waterdata._raise_for_non_200's catch-all for non-retryable 4xx stays a bareRuntimeError— load-bearing for the chunker's fatal-vs-resumable classifier; rooting it needs a dedicatedClientError(DataRetrievalError, RuntimeError)plus a chunker-classification review.🤖 Generated with Claude Code