Move engineering docs to spec/, resync with the implementation, and group the tree

[Alek99]

Summary

• move internal engineering documentation from docs/engineering/ to a root-level spec/ directory, and define that tree in CLAUDE.md as the source of truth that must stay synchronized with the implementation
• resync the tree with the code: ~100 claims that no longer matched shipped behavior
• add coverage for three surfaces that had no specification at all
• group the 15 top-level documents into api/, design/, matplotlib/, benchmarks/ and process/, and rewrite spec/README.md as a real index
• update every path consumer: build inputs, CI output paths, packaging rules, tests, generator scripts, and code comments

Why

The engineering documents are internal specifications rather than public docs-site content. Moving them to a top-level spec/ directory makes that role explicit.

But promoting a tree to "source of truth" only helps if it is true. Auditing all 24 documents against the implementation found the tree had drifted at PR #96 — five behavior PRs (#103, #105, #106, #108, #115) landed with no spec update. Three architecture documents described systems that no longer exist. Shipping the rename without that resync would have made the tree authoritative and wrong at the same time, so the first contributor to follow it would write wrong code.

Accuracy

Representative corrections, each verified against code:

• spec/design/rust-engine.md — ABI v31 → v36; "zero-crate cdylib" → png is a real dependency; pyramid ABI names (xy_pyramid_tile/_poll do not exist; shipped names are _count/compose/append); the panic shield was listed as pending but landed 29 ABI bumps ago as ffi_guard
• spec/design-dossier.md §3/§8 — described a Rust→WASM/Web-Worker/SharedArrayBuffer core; the shipped core is a C-ABI cdylib loaded by ctypes, as the same document's §32 already said. Reconciled, with the dropped WASM path recorded so the history is not lost
• spec/design/renderer-architecture.md — R2 claimed "No VAOs" via a _bindScalarAttr that does not exist; gl.createVertexArray() ships
• spec/matplotlib/compat.md — claimed the shim adds "+9% at 10k"; the guardrail it cites as proof says +60%
• spec/benchmarks/results.md — Plotly Scattergl at 10M appeared as both 54,064 ms and 33,907 ms from the same run; the derived headline multiplier was recomputed from the correct figure
• tier selection, ty check gate status, browser gate count, and the CI-enforced bundle budget corrected to match what CI actually does

Coverage

New documents for surfaces with zero prior coverage:

• spec/api/export.md — the static export surface shipped in Add export format parity and a unified export API (ENG-10447) #115 (to_image/write_image/write_images, five formats, Engine.auto routing). Records screen/export theme parity as an open gap rather than implying it is solved
• spec/api/interaction.md — interaction_config(), the event payloads, linking, and the gesture map (Add pan/zoom interaction switches for static-viewport charts #103/facet_chart: add opt-in linked panel interactions #105)
• spec/design/wire-protocol.md — the eight client↔Python request kinds and the packed-vs-split first-paint layout

Also added: the config.py threshold table (CLAUDE.md §28 promises every tier decision is recorded in the spec), full JS and Rust module inventories, and the hexbin centers-only wire contract that three renderers must expand identically — previously bound only by code comments, and once the cause of a CI-red payload regression.

Structure

spec/
  README.md              index of the whole tree
  design-dossier.md      entry point, cited by section number from code
  api/                   api-examples, chart-kind-contract, chart-roadmap,
                         styling, export, interaction
  design/                chart-grammar, lod-architecture, reflex-integration,
                         reflex-shaped-api, renderer-architecture, rust-engine,
                         wire-protocol
  matplotlib/            compat, compat-matrix, compat-changelog, shim-todo
  benchmarks/            results, metrics, methodology
  process/               contributing, production-readiness, security-audit
  assets/

spec/README.md previously listed only the two SVG assets, leaving 21 of 22 documents unreachable from any entry point; it is now a full index.

scripts/verify_sdist.py pinned only 5 top-level spec/*.md, so the sdist could have silently lost spec/design/ or spec/assets/ and still passed. It now asserts markdown under every subdirectory and both SVG snapshots.

Validation

Run locally on this branch:

• 1,908 tests pass, 78 skipped (tests/ and tests/pyplot)
• ruff 0.15.8 format --check and check both clean
• sync_matplotlib_compat.py --check exits 0 after the generator was repointed to the new location and link depth
• verify_sdist.py passes against a fresh sdist; the sdist contains all 27 spec/ files, verified by diffing its member list against the working tree in both directions
• check_claim_guardrails.py and verify_ci_workflow.py exit 0
• every markdown link and anchor in spec/ resolves (170 targets, checked with test -e from each containing file); zero repo-wide hits for any old spec path or for docs/engineering

Not verified locally: the benchmark regression gate. check_regressions.py needs fresh bench JSONs from a built engine plus Chrome/CDP, which do not run in this environment — CI has to confirm it. The --emit-md target and artifact upload path in ci.yml were repointed to spec/benchmarks/metrics.md and cross-checked against verify_ci_workflow.py's assertions, and a repo-wide grep confirms no writer still targets the old path, but the gate itself is unproven until CI runs.

[codspeed-hq]

Merging this PR will not alter performance

✅ 94 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing agent/move-engineering-docs-to-spec (5ebcad3) with main (914a327)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[masenf]

doesn't seem like this is useful now that spec is in the root

[masenf]

lgtm

[greptile-apps]

Greptile Summary

This PR moves internal engineering documentation into a root-level specification tree. The main changes are:

• Reorganizes engineering documents under spec/ by subject.
• Updates repository links, generated-document paths, and code comments.
• Updates CI benchmark outputs and source-distribution packaging.
• Adds checks for local links, specification layout, and benchmark content.

Confidence Score: 5/5

This looks safe to merge.

No blocking issues found in the changed code.

T-Rex Logs

What T-Rex did

• A fresh HEAD build and the verifier were run, and both exited with code 0.
• The preserved relocated-spec archive was independently verified by scripts/verify_sdist.py and passed, SHA-256 is 58cdf048e56241fb80313b27fbd07b9d6f1400f945eac4d0791c9ea71a612958.
• Inspection showed the relocated tree survived packaging, with 27 spec files now included in HEAD in the relocated grouped layout.
• Compared to HEAD^, the old layout packaged 24 files, while the relocated grouped layout now packages 27 files.

_{Ran code and verified through T-Rex}

Important Files Changed

Filename	Overview
pyproject.toml	Includes the relocated root-level specification tree in source distributions.
scripts/verify_sdist.py	Updates required specification paths and validates the new directory and asset layout.
.github/workflows/ci.yml	Moves benchmark metric generation and artifact upload to spec/benchmarks/metrics.md.
scripts/sync_matplotlib_compat.py	Moves generated compatibility output and updates relative corpus links for the deeper directory.
docs/app/xy_docs/config.py	Removes the obsolete exclusion for engineering documents after they moved outside the public docs tree.
scripts/check_regressions.py	Uses the new benchmark specification path for generated regression metrics.
tests/test_verify_sdist.py	Covers required files, subdirectories, and assets in the relocated specification tree.
tests/test_docs_examples.py	Updates documentation checks and local-link expectations to the new paths.

_{Reviews (1): Last reviewed commit: "Resync spec/ with the implementation and..." | Re-trigger Greptile}