Skip to content

test: catch models that silently drop MLB API values, and fix the 50 remaining fields - #249

Closed
Mattsface wants to merge 7 commits into
mainfrom
cursor/add-model-alias-drift-tests-81eb
Closed

test: catch models that silently drop MLB API values, and fix the 50 remaining fields#249
Mattsface wants to merge 7 commits into
mainfrom
cursor/add-model-alias-drift-tests-81eb

Conversation

@Mattsface

@Mattsface Mattsface commented Jul 31, 2026

Copy link
Copy Markdown
Member

Why

MLBBaseModel is configured extra="ignore", so a field whose alias does not match the key MLB sends is not a validation error — the value is discarded and the field keeps its None default forever. Nothing raises, nothing logs.

That is how issue #246 shipped in v0.7.1. When the data adapter stopped lowercasing every response key and moved to a camelCase alias generator, the generator only capitalised across underscores, so every field whose name is a single lowercase token kept expecting its own name. strikeouts expects strikeouts, MLB sends strikeOuts. Same for putouts, walkoffs, nickname, timezone, roundrobin, and the ax/ay/az pitch coordinates.

The important part is that the full suite was green the whole time. The stats tests assert on the response envelope (group, type, total_splits == len(splits)) and never on a stat value, so 62 permanently-None fields produced 191 passing tests. #244 and #247 fixed 12 of those fields; this PR adds the checks that would have caught all of them, and fixes what was left.

What

Tests (the point of the PR). Two layers, both driven by tests/alias_audit.py:

  • tests/test_model_alias_coverage.py runs offline against payloads recorded from the real API. It asserts that every recorded key naming a declared field actually reaches that field, that no field anywhere in the package expects a casing MLB does not use, and that each field named in the bug reports is populated.
  • tests/external_tests/models/test_alias_drift_live.py repeats those checks against the live API, so a rename on MLB's side surfaces even if nobody re-records, and flags fixtures that have gone stale.

Fixtures live in tests/fixtures/ and are regenerated with python tests/tools/record_api_fixtures.py, the only part that touches the network. It crawls 28 endpoints and captures 918 distinct keys.

Choosing the endpoints turned out to matter more than the checking logic. The first crawl took fielding stats from an outfielder, a person record from a living player and a game feed from before ABS challenges existed, so entire groups of keys were never observed and the guard stayed silent on them. The crawl now includes a catcher's fielding split, the outsAboveAverage stat type, a deceased player and a recent game feed, which took confirmed coverage of model fields from 96% to 98%. A floor test keeps that from silently regressing if a re-record fails.

Fixes needed to make those tests pass. 50 field aliases across pitching and fielding splits, sabermetrics, Person and its subclasses, Season, Venue, standings and team records, schedule and game data, and pitch coordinates. Two spellings cannot be derived from any rule and are pinned with a comment: MLB returns offseasonStartDate alongside offSeasonEndDate, and AdvancedPitchingSplit's strikeouts-to-walks ratio arrives as strikesoutsToWalks. That last one accepts both spellings via AliasChoices in case MLB ever corrects it.

Three further bugs the coverage work surfaced, each in its own commit:

  • get_venue and get_venues set params['hydrate'] to a Python list, which requests encodes as repeated hydrate query params. MLB answers those without hydrating, so Venue.location and Venue.timezone were None regardless of aliases. The existing venue test asserted hasattr, which passes on a pydantic model whether or not a value arrived.
  • SimpleFieldingSplit.catcher_era read catcherEra; MLB sends catcherERA. Only visible on a catcher's fielding split.
  • get_player_stats(stats=['outsAboveAverage']) raised ValidationError with 17 missing fields for every player. OutsAboveAverage declared the statistics directly on the split, but MLB nests them under stat like every other split type. Now nested in an OutsAboveAverageSplit, matching PitchArsenal, with the fields optional and fieldingRunsPrevented added.

CI. All three workflows ran pytest tests/external_tests/ only, so tests/test_base_model.py and tests/test_gamepk_validation_regressions.py have never executed on a push and the new offline tests would not have either. They now run pytest tests/.

Commits are ordered so the story is reviewable: the first test commit fails on main and reports exactly the fields that are broken, and the following commits fix them.

Tests

poetry run pytest tests/ goes from 191 passed to 294 passed, 2 skipped, including the live-API checks. CI passes on 3.10, 3.11 and 3.12.

Every alias in this PR was read off a live response rather than inferred — all 32 distinct alias strings introduced are present in the recorded vocabulary. Verified end to end against the live API:

Soto 2025 hitting:  strikeouts=137 groundouts=156 airouts=139 go/ao=1.12
Skubal pitching:    strikeouts=241 groundouts=160 airouts=168
Skubal advanced:    flyouts=96 lineouts=36 popouts=36 flyball_pct=.273 k/bb=7.30
Soto fielding:      putouts=249
Realmuto fielding:  catcher_era=3.76
Soto outsAboveAvg:  attempts=221 back=8.0 in=9.0 runs_prevented=16.0   (previously raised)
Person.nickname:    'Soto Pacheco'
Venue.timezone:     id='America/New_York' offset=-4 tz='EDT'

Also mutation-tested the guard: reverting SimpleFieldingSplit.putouts to its broken alias fails three tests with a message naming the field, the alias it expects, and the key the API sends.

Known blind spot. 50 of 2787 model fields still never appear in the crawl, because MLB only sends them in situations that cannot be requested on demand — resumed and rescheduled games, ejections, injury play events, delay durations — plus Stat.total_splits, which mlb_module computes rather than reads. Those fields are unverified either way; the guard reports a field only when the API demonstrably uses the key, so they stay quiet rather than producing false positives.

Risk and impact

Risk level: Normal

The alias corrections are mechanical and additive — fields that were unreachable now populate, and no field name, type or default changes, so existing attribute access keeps working. The one behavioural change users could notice is that code branching on these values being None will now take the other branch, which is the intended fix.

OutsAboveAverage is the only shape change: its statistics move from the split to split.stat. Nothing can depend on the old shape, because constructing it from a real response has always raised.

The venue hydrate change alters an outgoing request parameter, covered by both a live assertion and an offline assertion on the encoded value.

If something does go wrong it is contained to reading a field that was already unusable, and every alias change can be reverted individually.

Not included

Issue #245 (schedule requiring a date) is a separate bug in get_schedule / get_scheduled_games_by_date and is left out to keep this reviewable. Open PR #241 (str to float stat conversion) touches several of the same files, so it will need a rebase whichever lands second.

Open in Web Open in Cursor 

cursoragent and others added 7 commits July 31, 2026 14:28
MLBBaseModel ignores unknown keys, so a field whose alias does not match the
API exactly parses as None instead of raising. Issue #246 shipped that way in
v0.7.1 and survived a fully green test run, because the stats tests assert on
the response envelope and never on a stat value.

Adds two layers of checking:

- tests/test_model_alias_coverage.py runs offline against payloads recorded
  from the real API, asserting that every recorded key naming a declared field
  actually reaches it, and that no field in the package expects a casing the
  API does not use.
- tests/external_tests/models/test_alias_drift_live.py repeats those checks
  against the live API so renames on MLB's side surface without a re-record.

Fixtures are regenerated with tests/tools/record_api_fixtures.py.

These tests fail on main; the following commit fixes the fields they report.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
Completes the fix started in #244 and #247. The camelCase alias generator added
in v0.7.1 only capitalises across underscores, so every field whose name is a
single lowercase token kept expecting its own name: strikeouts, putouts,
walkoffs, nickname, timezone. MLB sends strikeOuts, putOuts, walkOffs, nickName
and timeZone, and because MLBBaseModel ignores unknown keys those values were
discarded rather than raising.

Corrects the 50 remaining fields, covering pitching and fielding splits,
sabermetrics, Person and its subclasses, Season, Venue, standings and team
records, schedule and game data, and pitch coordinates.

Two spellings could not be derived from any rule, so they are pinned with a
comment: MLB returns offseasonStartDate alongside offSeasonEndDate, and
AdvancedPitchingSplit's strikeouts-to-walks ratio arrives as
'strikesoutsToWalks'. The latter accepts both spellings via AliasChoices in
case MLB corrects the typo.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
get_venue and get_venues set params['hydrate'] to a list, which requests encodes
as repeated hydrate query params. MLB answers those without hydrating, so
Venue.location and Venue.timezone were always None no matter what the aliases
said.

The existing venue test only checked hasattr, which passes on a pydantic model
whether or not a value arrived, so this was invisible. Adds assertions on the
hydrated values and on the encoded parameter.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
CI ran only tests/external_tests/, so tests/test_base_model.py and
tests/test_gamepk_validation_regressions.py have never executed on a push, and
the new alias coverage tests would not have either.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
The first crawl took fielding stats from an outfielder, a person record from a
living player and a game feed from before ABS challenges existed, so whole
groups of keys were never observed and the guard stayed silent on them. Adds a
catcher's fielding split, the outsAboveAverage stat type, a deceased player and
a recent game feed, taking confirmed coverage of model fields from 96% to 98%
(918 distinct keys, up from 835).

Also asserts the recorded vocabulary still covers most of the model surface, so
a failed or truncated re-record cannot quietly disable these checks.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
SimpleFieldingSplit.catcher_era read 'catcherEra'. The field only appears on a
catcher's fielding split, which the earlier crawl never requested.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
get_player_stats(stats=['outsAboveAverage']) raised ValidationError with 17
missing fields for every player. OutsAboveAverage declared the statistics
directly on the split, but MLB nests them under 'stat' like every other split
type, so none of them were ever found and all were required.

Moves them to an OutsAboveAverageSplit that the split holds as stat, matching
PitchArsenal and the other Split subclasses, and makes them optional since this
stat type returns zeroes or nothing at all for most players. Adds
fieldingRunsPrevented, which the response includes but the model omitted.

The float widening is deliberate: the API returns values such as 8.0.

Co-authored-by: Matthew Spah <spahmatthew@gmail.com>
@Mattsface Mattsface closed this Aug 1, 2026
@Mattsface
Mattsface deleted the cursor/add-model-alias-drift-tests-81eb branch August 1, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants