Test coverage plan: the unit suite has never caught a bug

[solderzzc]

Tracking issue for test-coverage work, derived from the nine defects found during the #108/#110/#112 fix cycle rather than from a generic coverage target.

The observation

Sorting those nine defects by what actually caught them:

Caught by	Defects
Loading a real checkpoint	#118 (OptiQ norm shift), #120 (Gemma 4 KV-shared), the b674 e2b regression, per_layer_model_projection quantization
Code review	stop-sequence leak (#126), a delete-guard bypass, CLI cache-root mismatch, listed-but-not-loadable models
The unit suite	none

~250 unit tests exist and every one of them was written after its bug was already known. They lock in regressions, which is worth having, but they have never surfaced a defect. The defects lived in two places the suite does not reach: weight-and-config-shape assumptions, which only a real checkpoint exercises, and control flow, which only a reader caught.

Tier 1 — Architecture smoke matrix

Load + generate one token per checkpoint shape, not per model name. Shapes, and the defect each covers:

Shape	Representative	Covers
KV-shared, weights present	gemma-4-e2b-it-4bit	the b674 regression — landed in #127
KV-shared, weights absent	gemma-4-e4b-it-4bit	#120
Mixed precision + files outside the weight index	Qwen3.6-27B-OptiQ-4bit	#118
MoE with nested config	Qwen3.6-35B-A3B-4bit	#112
Dense baseline	Qwen2.5-0.5B-Instruct-4bit	—

Split by size so the cost is bounded: small shapes on every PR, the 19–20 GB ones nightly or only on submodule-bump PRs.

Tier 2 — API contract tests

One tiny model, asserting server behaviour end to end: stop sequences (including one split across token chunks), streaming deltas, the reasoning_content / content split, the two-round tool-call flow, JSON mode. This is the class review caught and the suite cannot see, and #126 is currently open with no scaffolding to verify a fix against.

Tier 3 — Make the local suite runnable

A bare swift test aborts on the second test with Failed to load the default metallib — locally about 1 of 250 tests runs. CI is unaffected because it installs the metallib explicitly (ci.yml, "Install MLX Metal library"), so this is a contributor trap: a local run looks green while testing almost nothing. Needs a test bootstrap step or a documented make test.

Tier 4 — Discipline

• Red-check every new test. This cycle produced a deletion-safety test that never called delete() and passed against the bug it claimed to cover.
• Turn on --enable-code-coverage to find blind spots objectively. Known example: MLXVLM/Models/Gemma4.swift has zero tests despite being where fix: Gemma 4 review follow-ups — assistant-case parity and KV-shared tests mlx-swift-lm#45's changes landed.

Open questions

• CI-minute budget per PR, which sets the small/nightly split for Tier 1.
• Whether large checkpoints live in a persistent cache or are fetched per run.