Skip to content

refactor(errors)!: unify request failures under a DataRetrievalError taxonomy#313

Merged
thodson-usgs merged 1 commit into
DOI-USGS:mainfrom
thodson-usgs:refactor/unify-http-error-handling
Jun 3, 2026
Merged

refactor(errors)!: unify request failures under a DataRetrievalError taxonomy#313
thodson-usgs merged 1 commit into
DOI-USGS:mainfrom
thodson-usgs:refactor/unify-http-error-handling

Conversation

@thodson-usgs
Copy link
Copy Markdown
Collaborator

@thodson-usgs thodson-usgs commented Jun 1, 2026

Problem

An HTTP failure surfaced as a different exception type depending on which module made the request, so there was no single except for "any dataretrieval request failure":

Path Raised on failure (before)
legacy query() (wqp, nwis, ngwmn, nldi) ValueError, or NoSitesError(Exception)
waterdata RateLimited / ServiceUnavailable (RuntimeError), RequestTooLarge (ValueError), ChunkInterrupted (RuntimeError)
nadp, streamstats bare httpx.HTTPStatusError

The same HTTP condition also produced near-homonym twins — ServiceUnavailableError vs ServiceUnavailable, RequestTooLargeError vs RequestTooLarge — distinguished only by an -Error suffix that silently encoded which code path raised it.

Change

Add dataretrieval.exceptions (dependency-free; every type re-exported at top level as dataretrieval.<Name>), rooted at DataRetrievalError, with two intermediate bases that name the axes a caller actually reasons about:

DataRetrievalError(Exception)
├─ BadRequestError(…, ValueError)        # 400
├─ NotFoundError(…, ValueError)          # 404
├─ RequestTooLarge(…, ValueError)        # base: the request is too large to satisfy
│   ├─ URLTooLong                        #   one request the server rejected (HTTP 414 / client-side)
│   └─ Unchunkable                       #   the chunker can't split the call small enough
├─ NoSitesError                          # empty result
└─ TransientError(…, RuntimeError)       # base: temporary failure worth a retry (carries retry_after)
    ├─ RateLimited                       #   429
    └─ ServiceUnavailable                #   5xx
  • One type per condition, shared by both paths — a 5xx raises ServiceUnavailable and a too-long URL raises a RequestTooLarge subclass whether the request came from the legacy query() path or the Water Data chunker.
  • Catch a whole familyexcept RequestTooLarge covers 414 + unchunkable; except TransientError covers any retryable failure; the chunker's retry check collapses to a single isinstance(exc, TransientError).
  • Built-in compatibility by kind — fatal client errors are also ValueError, transient transport errors also RuntimeError, so existing except ValueError / except RuntimeError keep working.
  • query()'s inline status ladder is extracted into a reusable _raise_for_status(); NoSitesError now subclasses DataRetrievalError (was Exception).
try:
    df, md = dataretrieval.wqp.get_results(...)
except dataretrieval.TransientError as e:
    time.sleep(e.retry_after or 1); ...   # handle just the retryable ones
except dataretrieval.DataRetrievalError:
    ...                                   # any failure, any module

Breaking changes

  • The legacy query() path raises typed errors instead of ad-hoc ValueErrors (400 → BadRequestError, 404 → NotFoundError, 414 / over-long URL → URLTooLong).
  • A 5xx on the legacy query() path now raises ServiceUnavailable, a RuntimeError (previously a ValueError) — a transient server failure is a runtime condition, not a bad value.
  • The Water Data chunker's planner-floor error is Unchunkable (a RequestTooLarge subclass).
  • Import the transport types / bases from dataretrieval / dataretrieval.exceptions, not from dataretrieval.waterdata.chunking.

The root catch (except DataRetrievalError) and ValueError / RuntimeError compatibility (by kind) are preserved.

Verification

  • 477 passed / 2 skipped; ruff clean.
  • Live API (legacy query() path): 404 → NotFoundError, 400 → BadRequestError, an over-long URL → URLTooLong (all DataRetrievalError + ValueError); a 200 is unaffected.
  • All 21 example notebooks execute end-to-end against the live API (227/227 code cells, 0 errors), exercising the modern waterdata path.

Out of scope (proposed follow-ups)

  • nadp / streamstats still raise httpx.HTTPStatusError — a one-line swap each to _raise_for_status, but it changes their public error type.
  • nldi's manual non-200 ValueError and nwis._parse_json_or_raise's HTML-detection ValueError aren't rooted yet.
  • waterdata._raise_for_non_200's catch-all for non-retryable 4xx stays a bare RuntimeError — load-bearing for the chunker's fatal-vs-resumable classifier; rooting it needs a dedicated ClientError(DataRetrievalError, RuntimeError) plus a chunker-classification review.

🤖 Generated with Claude Code

@thodson-usgs thodson-usgs force-pushed the refactor/unify-http-error-handling branch from 2afd2d7 to 9801177 Compare June 1, 2026 17:05
thodson-usgs added a commit to thodson-usgs/dataretrieval-python that referenced this pull request Jun 2, 2026
…ils/ package

## Why

`dataretrieval/waterdata/utils.py` had grown to 2033 LOC spanning ~6 unrelated
domains -- request building, response parsing, result finalization,
pagination/async, stats post-processing, and validation -- plus constants and
the public engines. It was the package's one genuine god-module. (An
architecture review found the package's OO is otherwise appropriate, so this is
a modularization, not an OO-pattern refactor.)

## What

Convert `utils.py` into a `utils/` package: the public surface stays in
`utils/__init__.py` (a thin facade) and the implementation is split across six
cohesive submodules, moving every definition verbatim (no signature/logic
changes):

| submodule | holds |
|---|---|
| `utils/constants.py` | URLs, `_OUTPUT_ID_BY_SERVICE`, regexes, param sets (dependency-free) |
| `utils/http.py` | headers, `_error_body`, `_raise_for_non_200`, retry-after |
| `utils/validate.py` | arg normalization/validation (`_get_args`, `_check_*`) |
| `utils/requests.py` | request building (`_construct_api_requests`, CQL2, dates) |
| `utils/responses.py` | geometry-agnostic parsing / finalization / stats shaping |
| `utils/engine.py` | pagination/async driver (`_paginate`, `_run_sync`, ...) |

`utils/__init__.py` re-exports the internal API (explicit `__all__`, 56 names),
so every existing `from dataretrieval.waterdata.utils import ...` and
`mock.patch("dataretrieval.waterdata.utils.<name>")` keeps working -- no import
sites or tests were touched. `dataretrieval.waterdata.utils` resolves to the
package's `__init__`, so the import path is unchanged from when it was a module.

Seven functions remain physically defined in `utils/__init__.py`
(`get_ogc_data`, `_fetch_once`, `get_stats_data`, `_get_resp_data`,
`_ogc_parse_response`, `_walk_pages`, `_handle_stats_nesting`) because the test
suite monkeypatches them (or `gpd`) by their `dataretrieval.waterdata.utils.*`
name, and a function's global lookups resolve in its defining module. The
geopandas probe stays with them, and the pagination logger keeps the name
`dataretrieval.waterdata.utils` (a caplog test pins it). These could later move
to the `engine`/`responses` submodules -- which do not import the package, so
there is no cycle -- but that requires re-targeting the test patches; left as a
follow-up.

## Behavior-preserving

- 56 top-level definitions moved verbatim -- none lost, none duplicated.
- 469 tests pass, 2 skipped; ruff clean; submodules import without cycles
  (`constants` <- `http`/`validate` <- `requests`/`responses` <- `engine` <-
  `__init__`); `chunking.py` untouched.

## Note

Overlaps with the error-taxonomy (DOI-USGS#313) and namespace (DOI-USGS#315) PRs on `waterdata/`
imports -- sequence on merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@thodson-usgs thodson-usgs changed the title refactor(errors): unify request errors under a DataRetrievalError root refactor(errors)!: unify request failures under a DataRetrievalError taxonomy Jun 2, 2026
@thodson-usgs thodson-usgs force-pushed the refactor/unify-http-error-handling branch from 12cd148 to 5bc68a6 Compare June 2, 2026 19:16
…taxonomy

Before, an HTTP failure surfaced as a different exception type depending on
which module made the request -- a ValueError (or bare Exception) on the legacy
query() path, RuntimeError-based types on the waterdata path, a bare
httpx.HTTPStatusError elsewhere -- so there was no single `except` for "any
dataretrieval request failure".

Introduce dataretrieval/exceptions.py (dependency-free, re-exported at top level
as dataretrieval.<Name>), rooted at DataRetrievalError, with two intermediate
bases that name the axes a caller reasons about:

  DataRetrievalError(Exception)
  |- BadRequestError(.., ValueError)     # 400
  |- NotFoundError(.., ValueError)       # 404
  |- RequestTooLarge(.., ValueError)     # base: request too large to satisfy
  |   |- URLTooLong                      #   414 / client-side URL reject
  |   '- Unchunkable                     #   chunker planner floor
  |- NoSitesError                        # empty result
  '- TransientError(.., RuntimeError)    # base: retryable; carries retry_after
      |- RateLimited                     #   429
      '- ServiceUnavailable              #   5xx (both paths)

- One type per condition, raised by both the legacy query() path and the Water
  Data chunker. Callers can catch a whole family (`except RequestTooLarge` /
  `except TransientError`); the chunker's retry check is a single
  isinstance(exc, TransientError).
- query()'s inline status ladder is extracted into a reusable _raise_for_status().
- NoSitesError now subclasses DataRetrievalError (was Exception).
- Built-in compatibility by kind: fatal client errors are also ValueError,
  transient transport errors also RuntimeError, so existing `except ValueError`
  / `except RuntimeError` handlers keep working.

BREAKING CHANGES
- The legacy query() path raises typed errors instead of ad-hoc ValueErrors
  (400 -> BadRequestError, 404 -> NotFoundError, 414/over-long URL -> URLTooLong).
- A 5xx on the legacy query() path now raises ServiceUnavailable, a RuntimeError
  (was a ValueError): a transient server failure is a runtime condition, not a
  bad value.
- The Water Data chunker's planner-floor error is Unchunkable (a RequestTooLarge
  subclass).
- Import the transport types/bases from dataretrieval / dataretrieval.exceptions,
  not from dataretrieval.waterdata.chunking.

Verified: 477 passed / 2 skipped, ruff clean; live API spot checks (404/400/
over-long URL raise the typed errors, 200 unaffected); all 21 example notebooks
execute end-to-end against the live API (227/227 cells, 0 errors).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thodson-usgs thodson-usgs force-pushed the refactor/unify-http-error-handling branch from 5bc68a6 to d46de2b Compare June 2, 2026 19:35
@thodson-usgs thodson-usgs marked this pull request as ready for review June 3, 2026 00:50
@thodson-usgs thodson-usgs merged commit ecf2833 into DOI-USGS:main Jun 3, 2026
9 checks passed
@thodson-usgs thodson-usgs deleted the refactor/unify-http-error-handling branch June 3, 2026 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant