Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 66 additions & 32 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,101 @@
# views-frames

The VIEWS platform's **data-contract layer**: small, stable, abstract, immutable
array+identifier value objects (`FeatureFrame`, `PredictionFrame`, `TargetFrame`,
and anticipated siblings) at the **root of the platform dependency DAG**. numpy
only; depends on nothing internal; every other repo depends *toward* it.
array+identifier value objects (`PredictionFrame`, `TargetFrame`, `FeatureFrame`)
at the **root of the platform dependency DAG**, plus two sibling operation packages
in the same wheel. numpy only; depends on nothing internal; every other repo
depends *toward* it.

> **Status:** **implemented — v0.1.0** (Epic 2). `src/views_frames/` realises the
> contract (index, frames, io, conformance suite). Consumer adoption (re-export
> shims, pandas migration) is Epic 3.
> **Status:** **released — v1.8.x on PyPI**, public API **frozen since v1.0.0**
> (ADR-018; everything after is additive, `CONFORMANCE_FLOOR` stays `1.0.0`).
> Consumers install `views-frames` and validate against the published conformance
> suite (`views_frames.conformance`, ADR-016). See `CHANGELOG.md` for the release
> history and `README.md` §status for the version chronicle.

## Architecture

**Two packages** under `src/` (datafactory multi-package pattern), strict one-way
dependency `views_frames_summarize → views_frames`, enforced by
`tests/test_import_enforcement.py`:
**Three packages** under `src/` (one wheel), strict one-way dependencies
`views_frames_summarize → views_frames` and `views_frames_reconcile → views_frames`
(siblings never import each other), enforced by `tests/test_import_enforcement.py`:

**`src/views_frames/`** — the pure data contract (numpy-only; depends on nothing):
**`src/views_frames/`** — the pure data contract (numpy-only; depends on nothing; frozen):

- `index.py` — `SpatioTemporalIndex` (`{time, unit, level}` + same-level numpy alignment).
- `index.py` — `SpatioTemporalIndex` (`{time, unit, level}`; same-level numpy alignment +
consumer-injected cross-level remap; identifier arrays write-protected).
- `spatial_level.py` — `SpatialLevel` (cm/pgm identifier vocabulary; labels only).
- `protocols.py` — `Frame` / `SpatioTemporalIndexed` / `Sampled` (`sample_count`/`is_sample` only) / `Persistable`.
- `protocols.py` — `Frame` / `SpatioTemporalIndexed` / `Sampled` (`sample_count`/`is_sample`
only) / `Persistable` (four small segregated protocols).
- `metadata.py` — `FrameMetadata` (typed, frozen, generic-only provenance; ADR-020).
- `_validation.py` — shared construction-time invariants.
- `feature_frame.py`, `prediction_frame.py`, `target_frame.py` — sibling frames (no shared base; ADR-011 Option C).
- `io/` — `npz` (native) + `arrow` (flat-columnar). **The only place `pyarrow` may be imported.**
- `feature_frame.py`, `prediction_frame.py`, `target_frame.py` — sibling frames
`(N,F,S)` / `(N,S)` / `(N,1)` (no shared base; ADR-011 Option C).
- `io/` — `npz` (native, mmap-capable) + `arrow` (flat-columnar parquet codec,
module-level by decision D-11). **The only place `pyarrow` may be imported.**
- `conformance/` — the published suite consumers run in *their* CI
(`assert_frame_contract`, `assert_frame_envelope`, alignment laws; `CONFORMANCE_FLOOR`).

**`src/views_frames_summarize/`** — sample-axis posterior summarization *over* frames
(ADR-017; numpy-only; depends on `views_frames`). Point estimates
(`collapse(frame, reducer)`, `map_estimate`) return a `(N,…,1)` frame; intervals
(`hdi`, `quantiles`) return arrays aligned to the frame's index. **Never** owns IO,
domain data, scoring, or reconciliation.
(ADR-017; numpy-only; depends on `views_frames`). Point estimates (`collapse`,
`map_estimate`, `tower_point`) return `(N,…,1)` frames; intervals/arrays (`hdi`,
`quantiles`, `hdi_tower`, `exceedance`, `expected_shortfall`, `bimodality`) return
index-aligned arrays; `aggregate_distributions[_arrays]` sums sample distributions
across levels (joint sampling). Fail-loud config for the tower family (ADR-019).
**Never** owns IO, domain data, scoring, or reconciliation.

**`src/views_frames_reconcile/`** — top-down proportional reconciliation of pgm grid
forecasts to cm country totals (ADR-023; numpy-only; depends on `views_frames`).
`ReconciliationModule(map_keys, map_vals)` with the geography **injected** (never
fetched); `reconcile` / `reconcile_result` (the latter reports the
`point-broadcast`/`aligned-draws` **mode** — returned, never stamped on the leaf
header, D-12). The per-draw method is a documented approximation; the principled
joint upgrade is designed and deferred (ADR-024, register C-62).

## Tooling (uv + hatchling)

Always invoke via `uv run`:

```bash
uv sync # install deps + the package (editable)
uv run pytest # tests (incl. import-enforcement + falsification stubs)
uv run pytest # tests (incl. import-enforcement + falsification suites)
uv run ruff check . # lint
uv run ruff format . # format
uv run mypy src/ # type check (strict)
uv run mypy src/ # type check (strict; also check with --python 3.11 before pushing)
uv build # build wheel + sdist
```

CI additionally gates 100% line+branch coverage
(`uv run pytest --cov --cov-fail-under=100`) and a numpy-floor job.

## Design principles (the hard constraints)

1. **numpy only in the core.** Never import `pandas`, `polars`, `geopandas`,
`wandb`, `viewser`, `torch`, or any `views_*` package. `pyarrow` is allowed
`wandb`, `viewser`, `torch`, or any foreign `views_*` package. `pyarrow` is allowed
*only* under `io/`. Enforced by `tests/test_import_enforcement.py` (ADR-002).
2. **Immutable value objects.** Operations return new frames; structural ops share
the buffer (zero-copy); only reductions allocate (ADR-013, register C-07).
3. **Fail loud at construction.** Invariants raise `ValueError`/`TypeError`; the
guarantee is *structural*, not temporal (`time` is opaque; register C-11).
the buffer (zero-copy); only reductions allocate (register C-07). Enforced for the
*index*; **by convention** for the value buffer (writeable on purpose to preserve
zero-copy/mmap — mutating `.values` in place is unsupported; ADR-025, C-66 rider).
3. **Fail loud.** Invariants raise `ValueError`/`TypeError` at construction and at
every validation guard; the guarantee is *structural*, not temporal (`time` is
opaque; register C-11).
4. **No shared frame base.** Frames are separate siblings (ADR-011); cm/pgm is a
`SpatialLevel` *value*, never a class axis.
5. **No domain data.** Cross-level cm↔pgm alignment is a protocol with a
consumer-injected mapping; the leaf never embeds/fetches it (ADR-014).
6. **One concept per file**; explicit `__init__.py` re-exports (no `import *`).
5. **No domain data.** Cross-level cm↔pgm alignment takes a consumer-injected
mapping; the leaf and siblings never embed/fetch it (ADR-014/ADR-023).
6. **One concept per file** (test-enforced: ≤1 public class per module); explicit
`__init__.py` re-exports (no `import *`).
7. **Frozen surface + WET before DRY.** The v1 public surface only grows additively
(removal/tightening = MAJOR + cross-repo coordinated bump, GOVERNANCE.md);
deliberate duplication recorded in ADRs is a choice, not debt.

## Governance

Constitutional ADRs 000–010, project ADRs 011–016, CIC infrastructure, contributor
protocols, and standards live in `docs/`. The technical risk register is
maintained as an internal governance artifact. Run `bash docs/validate_docs.sh` to check
documentation consistency. Build *against* the README design bible — if code and
README disagree, reconcile before merging.
Constitutional ADRs 000–010, project ADRs 011–025, CICs for every non-trivial
surface (7 active incl. the package-level `Summarize.md` and `Reconcile.md`),
contributor protocols, and standards live in `docs/`. The technical risk register
(`reports/technical_risk_register.md`) is the curated concern/decision log. Run
`bash docs/validate_docs.sh` to check documentation consistency. Build *against*
the README design bible — if code and README disagree, reconcile before merging.
Releases: dev→main via **merge commit** (never squash — main carries squash release
commits dev lacks), then `gh release create vX.Y.Z` triggers the PyPI publish
(Trusted Publishing; see `docs/guides/publishing-to-pypi.md`).
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,18 @@
> containers (`FeatureFrame`, `PredictionFrame`, and their anticipated siblings)
> that every other repo depends on and that depends on nothing internal.
>
> **Status:** **v1.7.0 — frozen API** (frozen since v1.0.0, ADR-018; the v1.1 surface is
> **Status:** **v1.8.0 — frozen API, published to PyPI** (frozen since v1.0.0, ADR-018; the
> v1.1 surface is
> purely additive — the coherent posterior summary, ADR-019; v1.2.0 rebuilt the tower
> `outside-in`, C-44; v1.3.0 makes the tower summary distribution-agnostic — no magnitude
> zeroing by default, register C-45; v1.4.0 adds generic provenance to `FrameMetadata`
> (`run_id`/`data_version`) and publishes the shared `assert_frame_envelope` checker,
> ADR-020; v1.5.0 adds the threshold **exceedance** estimator `P(Y > c)`, ADR-021; v1.6.0
> adds the worst-case **expected_shortfall** estimator, ADR-022; v1.7.0 adds a third
> sibling package **`views_frames_reconcile`** — forecast reconciliation, ADR-023). This
> sibling package **`views_frames_reconcile`** — forecast reconciliation, ADR-023; v1.8.0
> adds the native point-country broadcast + the self-describing reconciliation mode,
> the three showcase notebooks, and the frames-family hardening pass — Reconcile.md CIC,
> ADR-025 immutability-by-convention, the adversarial red-test batch). This
> README is the design
> bible; the contract it specifies is realised in `src/views_frames/` (index, frames,
> io, conformance suite) plus the `src/views_frames_summarize/` sibling package
Expand Down
11 changes: 11 additions & 0 deletions docs/ADRs/013_metadata_and_identifier_model.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,17 @@ the index aligns rows; the header describes the frame.

## Implementation Notes

> **As-built amendment (2026-07-02, four-axis audit / register C-70).** Two details landed
> differently from the notes below, both consistent with the decision's substance:
> (1) **`feature_names` is not a header field** — it is a `FeatureFrame` constructor
> argument/attribute (`feature_frame.py`), validated there against the feature axis
> (`len(feature_names) == values.shape[1]`) and serialized with the frame; `FrameMetadata`
> stays purely generic provenance (which ADR-020 later ratified as generic-*only*).
> (2) "Validated at construction" applies to the **frames and identifiers**, not the header
> fields themselves — `FrameMetadata` is a frozen dataclass of all-optional fields;
> `from_dict` tolerates unknown keys (forward-compatible). The typed-optional-extensible
> header and fixed `{time, unit}` identifiers — the decision's core — are implemented as decided.

- Model the header as a frozen dataclass (`FrameMetadata`) with all-optional fields + validation;
`feature_names` lives here for `FeatureFrame`.
- `_validation.py` enforces required identifiers `{time, unit}`; optional identifiers validated
Expand Down
20 changes: 12 additions & 8 deletions docs/ADRs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,17 @@
# ADR README and Governance Map — views-frames

This repository uses Architecture Decision Records (ADRs) to govern structural, semantic,
and operational behavior. Because `views-frames` is currently a **design bible** (a
README, consumer-review findings, design critiques, and falsification stubs — no `src/` yet), these
ADRs describe the **intended** architecture; when the leaf is stood up, code must conform
to them (and to the README, which is itself authoritative).
and operational behavior. The architecture these ADRs describe is **implemented and
released** (three packages under `src/`, public API frozen since v1.0.0); code must
conform to them (and to the README design bible, which is itself authoritative — if code
and README disagree, reconcile before merging).

ADRs are divided into:

1. **Constitutional ADRs (000–009)** — foundational architectural rules.
2. **Governance ADRs (010)** — the technical risk register.
3. **Project-Specific ADRs (011–016)** — the ratified contract decisions.
3. **Project-Specific ADRs (011–025)** — the ratified contract, estimator, sibling-package,
and freeze/immutability decisions.

---

Expand Down Expand Up @@ -50,10 +51,13 @@ Together, these define the invariant layer of the system.

---

## Project-Specific ADRs (011–016) — the ratified contract decisions
## Project-Specific ADRs (011–025) — the ratified project decisions

These ratify the six resolved decisions from the design bible (README §13a). All **Accepted**
2026-06-21; each cites the risk-register IDs it resolves.
**011–016** ratify the six founding contract decisions from the design bible (README §13a,
all Accepted 2026-06-21); **017–025** are the post-v1 decisions — sibling packages, the API
freeze, estimators, contract homes, and the immutability convention (Accepted 2026-06-22
through 2026-07; ADR-024 is a design-direction ADR with implementation deferred). Each cites
the risk-register IDs it resolves.

- **ADR-011** — [Twin-unification model (Option C)](011_twin_unification_option_c.md). Share only `SpatioTemporalIndex` + `_validation` + protocols + `io/`; relocate the frames as separate sibling classes; reject the `_BaseFrame` god-class (A); defer the composed header (B). Resolves D-03; relates to C-16, C-03.
- **ADR-012** — [Sample axis convention](012_sample_axis_convention.md). Always an explicit trailing axis (`S ≥ 1`); `is_sample = S > 1`; `collapse` reduces the trailing axis; one shape contract. Relates to C-02, C-16, C-17.
Expand Down
4 changes: 2 additions & 2 deletions docs/CICs/FeatureFrame.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

**Status:** Active
**Owner:** VIEWS platform maintainers
**Last reviewed:** 2026-06-24
**Last reviewed:** 2026-07-02
**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013

> Implemented in v0.1.0 (`src/views_frames/feature_frame.py`). This contract governs
Expand Down Expand Up @@ -59,7 +59,7 @@ Violations raise at construction (ADR-008).

## 5. Outputs and Side Effects

- New frames from operations; `save` writes `y_features.npy` + `identifiers.npz`
- New frames from operations; `save` writes `values.npy` + `identifiers.npz`
+ `feature_names`/`metadata` (via the frame-state round-trip contract).

---
Expand Down
11 changes: 7 additions & 4 deletions docs/CICs/PredictionFrame.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

**Status:** Active
**Owner:** VIEWS platform maintainers
**Last reviewed:** 2026-06-24
**Last reviewed:** 2026-07-02
**Related ADRs:** ADR-001, ADR-008, ADR-011, ADR-012, ADR-013, ADR-017

> Implemented in v0.1.0 (`src/views_frames/prediction_frame.py`), relocated from
Expand Down Expand Up @@ -61,16 +61,19 @@ Violations raise at construction (ADR-008) — never log-and-continue.

## 5. Outputs and Side Effects

- New frames from operations; `save` writes `y_pred.npy` + `identifiers.npz`
- New frames from operations; `save` writes `values.npy` + `identifiers.npz`
(+ header). No other side effects.

---

## 6. Failure Modes and Loudness

- Raises `TypeError` on non-`float32`/object-dtype values; `ValueError` on shape,
- Raises `ValueError` on object-dtype values (list-in-cell is banned), on shape,
length, or completeness violations, and on `reindex(other)` when `other` is not a
subset of this frame's index. The structural guarantee is **not temporal** (register C-11).
subset of this frame's index. Other numeric dtypes (e.g. float64) are **coerced**
to float32 by copy at construction — accepted, not rejected (the no-copy fast path
is float32-only, register C-07). The structural guarantee is **not temporal**
(register C-11).

---

Expand Down
11 changes: 6 additions & 5 deletions docs/CICs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,17 +49,18 @@ Contracts must be clear enough that:

---

## Status: infrastructure only (no contracts yet)
## Status: fully contracted

`views-frames` currently contains no code — it is a design bible (see `README.md`). This
directory holds the template (`cic_template.md`) and this README. **Contracts are authored
as each class is implemented**, when the twins are relocated and the leaf is stood up.
Every non-trivial surface across the three shipped packages (`views_frames`,
`views_frames_summarize`, `views_frames_reconcile`) is governed by an active CIC below.
New contracts are authored **with** the class/package that introduces them (the
`Reconcile.md` gap was the last, closed 2026-06-28, register C-64).

---

## Active Contracts

These CICs govern the classes implemented in v0.1.0 (`src/views_frames/`).
These CICs govern the shipped surface (`src/`, three packages, frozen since v1.0.0).

- `SpatioTemporalIndex.md` — the genuinely-reused alignment primitive; same-level logic owned,
cross-level mapping injected (ADR-014).
Expand Down
22 changes: 14 additions & 8 deletions docs/CICs/Reconcile.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,14 +255,20 @@ gate); these are the contract-bearing suites:
both-points, pre-tiled cm) — `TestReconciliationResult`.
- **Red (it fails loud):** `tests/test_reconciliation_validation.py` —
`validate_reconciliation_inputs` raises on wrong cm level, wrong grid level, sample-count
mismatch, time mismatch, and a missing country forecast; the `(M, 2)` map-keys guard
(`test_bad_mapping_shape_raises`). The conformance-suite **negative** (a deliberately
non-conforming impl/input makes `assert_reconcile_contract` raise) and the
`ReconciliationResult` frozen-ness assertion are the red gaps being closed alongside this
contract (register C-65 batch, epic #179 / S3).

Each §3 guarantee maps to a green/beige test; each §6 failure mode maps to a red test in
`test_reconciliation_validation.py`.
mismatch, time mismatch, a missing country forecast, and a missing
`(time, priogrid_gid)` mapping entry (`test_missing_mapping_entry_raises`); the `(M, 2)`
map-keys guard (`test_bad_mapping_shape_raises`). The conformance-suite **negative** (a
deliberately non-conforming impl makes `assert_reconcile_contract` raise) and the
`ReconciliationResult` frozen-ness assertion live in `test_reconcile_conformance.py` /
`test_reconciliation_e2e_parity.py` (the C-65 batch, epic #179 / S3). The
share-proportionality **law** (`test_shares_are_preserved_within_each_draw`) and the
2026-07 audit regressions — exact conservation for tiny nonzero sums, the negative-total
raise — are pinned in `test_reconciliation_parity.py` and
`test_falsification_safety_audit_2026_07.py` (register C-68/C-70).

Each §3 guarantee maps to a green/beige test; each §6 failure mode maps to a red test
(the validation guards in `test_reconciliation_validation.py`; the negative-total and
tiny-sum modes in the audit-regression file above).

---

Expand Down
7 changes: 5 additions & 2 deletions docs/CICs/SpatioTemporalIndex.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,11 @@ Assumptions that do not hold **must raise** at construction (ADR-009), never fal

## 6. Failure Modes and Loudness

- Raises `TypeError` if any identifier array is not integer dtype.
- Raises `ValueError` if arrays differ in length, contain NaN, or `level` is invalid.
- Raises `TypeError` if any identifier array is not integer dtype — which is also what
a NaN-carrying array hits (NaN forces a float dtype; integers cannot be NaN, so the
"no NaN" guarantee is enforced *via* the dtype check, not a separate `ValueError`).
- Raises `ValueError` if arrays differ in length; `TypeError` if `level` is not a
`SpatialLevel`.
- `cross_level_align` raises if called without an injected mapping — it must **never**
silently fetch or assume one (the defining boundary of ADR-014).
- Nothing fails silently.
Expand Down
2 changes: 1 addition & 1 deletion docs/CICs/Summarize.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

**Status:** Active
**Owner:** VIEWS platform maintainers
**Last reviewed:** 2026-06-24
**Last reviewed:** 2026-07-02
**Related ADRs:** ADR-002, ADR-008, ADR-009, ADR-011, ADR-012, ADR-014, ADR-017, ADR-019, ADR-021

> The `views_frames_summarize` package (functions, not a class). Implemented in
Expand Down
Loading
Loading