dfa1 · dfa1 · Jul 20, 2026 · Jul 20, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -22,6 +22,10 @@ Benchmark classes follow this: `JavaVsJni{Read,Write,Filter}Benchmark`,
 ## Module structure
 
 ```
+fsst    — io.github.dfa1.vortex.fsst: standalone FSST (Fast Static Symbol Table) string
+          compression algorithm — Symbol, Compressor/CompressorBuilder, Decompressor/Matcher.
+          Zero dependency on core/reader/writer/FFM-wire-concerns beyond the JDK's own
+          java.lang.foreign; writer/reader depend on it, not the other way around.
 core    — everything lives under `io.github.dfa1.vortex.core.*`:
           core.model    DType, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype
           core.io       IoBounds, PTypeIO, VortexFormat

diff --git a/TODO.md b/TODO.md
@@ -13,6 +13,24 @@
 - [ ] Run performance tests on other machines (I have access only to Apple M5)
 - [ ] **Vector API adoption** — see [ADR-0005](adr/0005-vector-api-adoption.md).
 
+### FSST follow-ups
+
+Out of scope for the #287 rewrite (which closed most of the `vortex-jni` gap with a scalar,
+branch-free algorithm — see [ADR-0022](adr/0022-fsst-module-extraction.md)):
+
+- [ ] **AVX512/SIMD compression kernel** — the paper measures a SIMD kernel as the fastest known
+  string compressor at that tier, but this project's scalar rewrite already narrowed the encode gap
+  to `vortex-jni` from 36x to 1.6x. Needs a Vector-API/JDK-incubator decision and its own ADR (cf.
+  [ADR-0005](adr/0005-vector-api-adoption.md)).
+- [ ] **True per-row lazy/random-access decompression** exploiting FSST's headline random-access
+  property — today `FsstEncodingDecoder.decode()` eagerly materializes the whole column up front
+  regardless of what is queried. Connects to [ADR-0010](adr/0010-lazy-decode.md) (Lazy decode) but
+  is a separate initiative.
+- [ ] **OptFSST** (2026 arXiv follow-up: DP-based training instead of greedy, ~4x slower training
+  for 7–17% better compression) — a documented future option, not adopted, since it moves off the
+  classic greedy-FSST speed/compression tradeoff this rewrite targets (matching what `vortex-jni`
+  itself uses).
+
 ## Security
 
 See [CLAUDE.md §Security contract](CLAUDE.md) for the invariant. Each entry below is either a

diff --git a/adr/0022-fsst-module-extraction.md b/adr/0022-fsst-module-extraction.md
@@ -0,0 +1,145 @@
+# ADR 0022: Extract FSST into a standalone module, ported faithfully from the paper
+
+- **Status:** Accepted
+- **Date:** 2026-07-20
+- **Deciders:** project maintainer
+- **Supersedes:** —
+- **Superseded by:** —
+- **Related:** [ADR 0005 — Vector API adoption](0005-vector-api-adoption.md), [ADR 0010 — Lazy
+  decode](0010-lazy-decode.md), [ADR 0017 — In-house FlatBuffers codegen](0017-in-house-flatbuffers-codegen.md)
+
+## Context
+
+`writer/FsstEncodingEncoder` and `reader/FsstEncodingDecoder` held a first-pass, simplified FSST
+(Fast Static Symbol Table) implementation. Comparing it against `vortex-jni` (issue #287) surfaced
+real gaps against both the original paper (Boncz/Neumann/Leis, "FSST: Fast Random Access String
+Compression", PVLDB 13(11), 2020) and the Rust reference (`spiraldb/fsst`, the crate the Rust
+`vortex-fsst` wire adapter is built on):
+
+- **Matching.** The old `longestMatch` probed symbol lengths 8 down to 1 in a per-position loop —
+  up to eight sequential open-addressing hash lookups per input byte, a variable-length loop body
+  that blocks JIT auto-vectorization (the exact shape CLAUDE.md's hot-loop rule forbids). The
+  paper's Algorithm 4 ("lossy perfect hashing", §5.1) resolves any match in O(1): a direct
+  65536-entry array for 0/1/2-byte symbols, and a small hash table for 3-8 byte symbols, combined
+  branch-free.
+- **Training.** The old bottom-up loop always compress-counted the *full* training sample on every
+  one of its five generations, ranked candidates by plain `count * length`, and never pruned
+  low-frequency noise. The real reference (`spiraldb/fsst`) additionally uses a growing
+  per-generation sample fraction (~6% → 100%), a per-generation min-count floor, an 8x gain boost
+  for single-byte candidates (to suppress escapes), and a final cost-based prune pass.
+- **Decode.** The old decoder copied a matched symbol's bytes one at a time in a per-byte loop
+  instead of the paper's Algorithm 1 trick — one unconditional 8-byte store, advancing the output
+  cursor by only the symbol's true length.
+
+A zero-dependency Java FSST implementation was searched for and does not exist (only the C
+reference `cwida/fsst`, the Rust `spiraldb/fsst`, and a Go port `axiomhq/fsst`). Per that search's
+own conclusion, the fix isolates FSST into its own module — mirroring how the Rust reference itself
+splits a pure algorithm crate (`spiraldb/fsst`, zero dependencies, no Vortex awareness) from
+vortex-rust's thin wire adapter (`vortex-fsst`) — so the compression algorithm can be iterated on
+and benchmarked without the Vortex wire-format "shell" in the way.
+
+During the rewrite's own review, a real data-corruption bug was found and fixed: the new
+branch-free matcher has no notion of "end of input" (unlike the old bounded loop), so a trained
+symbol whose trailing bytes happen to be zero could spuriously match zero-padded bytes past a short
+input's true end, silently appending garbage bytes to the decoded output. Confirmed with a concrete
+repro (a 3-byte input round-tripping to 8 bytes) before it reached a merged PR. Fixed by an explicit
+`pos + length <= end` guard at both call sites that feed the matcher a possibly-short remaining
+range (`Compressor.compress` and the training loop's compress-count step) — the paper's own
+guidance for a branch-free scalar kernel (§5.2) names exactly this bound as the alternative to a
+terminator byte (which is an AVX512-batch-kernel-specific optimization, out of scope here).
+
+## Decision
+
+Extract FSST into a new top-level Maven module, `fsst` (artifact `vortex-fsst`), with zero
+dependency on `core`/`reader`/`writer`:
+
+1. **Port the paper faithfully**, plus the real reference's engineering refinements: branch-free
+   O(1) matching (`ShortCodeTable` for 0/1/2-byte symbols, a 2048-slot lossy perfect hash table for
+   3-8 byte symbols — matching `spiraldb/fsst`'s L1D-cache-line-split sizing over the paper's
+   literal 4096, since that Rust crate is the actual `vortex-jni` comparison target); adaptive
+   byte-size-bounded sampling with a growing per-generation fraction, min-count pruning, the
+   single-byte gain boost, and the final cost-based prune during training; the unconditional-8-byte-
+   store decode trick.
+2. **Memory boundary: `MemorySegment`, not `ByteBuffer`.** `java.lang.foreign` is a standard JDK
+   module, not a third-party dependency — using it satisfies "zero deps" exactly as well as
+   `ByteBuffer` would, while staying consistent with every other module in this codebase (CLAUDE.md:
+   "Uses FFM (`MemorySegment`/`Arena`) — never JNI or `sun.misc.Unsafe`") and avoiding a dual-API
+   translation layer with its own parity-testing burden. The module's hot-path `compress`/
+   `decompress` methods take `MemorySegment` directly; plain `byte[]` overloads exist alongside them
+   for standalone use with zero FFM ceremony.
+3. **`writer`/`reader` become thin wire adapters.** `FsstEncodingEncoder`/`FsstEncodingDecoder` keep
+   all Vortex-specific plumbing (UTF-8 conversion, `ProtoFSSTMetadata`, `EncodeNode`/`EncodeResult`
+   assembly, `Arena` allocation) and delegate the algorithm itself to `CompressorBuilder`/
+   `Compressor`/`Decompressor`. The wire format (`vortex.fsst`) is unchanged — this is a pure
+   algorithm/performance rewrite, not a wire-format change. The one adapter-side subtlety: the
+   module numbers codes gain-descending internally (load-bearing for the hash table's
+   first-writer-wins collision rule), while the wire format requires length-sorted order
+   (`Compressor#codesSortedByLength()`), so the writer remaps every code the compressor emits
+   through the wire permutation before writing it out.
+
+## Consequences
+
+### Positive
+
+- Measured, real speedup (`JavaVsJniFsstBenchmark`, unmodified before and after — see the
+  `[Unreleased]` CHANGELOG entry for the full table): `javaFsstEncode` 0.085 → 1.848 ops/s (~21.7x
+  faster, closing the gap to `vortex-jni` from 36x slower to 1.6x slower); `javaFsstDecode` 5.243 →
+  27.137 ops/s (~5.2x faster, gap 6.5x → 1.3x).
+- The `fsst` module is independently testable and benchmarkable with zero Vortex file-format
+  scaffolding in the loop (`FsstEncodingEncoderTest`'s wire-format tests are unaffected; new
+  `fsst`-module tests exercise the algorithm directly) — the issue's own stated motivation for the
+  extraction.
+- A golden test (`PaperFigure2Test`) pins the paper's own worked example against a hand-built
+  (not trained) symbol table — an external, human-verifiable fixed point independent of this
+  project's training heuristics.
+
+### Negative
+
+- Real rework: five new PRs' worth of module surface (`Symbol`, `Matcher`/`ShortCodeTable`/
+  `LossyPerfectHashTable`, `Sample`/`TrainingGeneration`/`CompressorBuilder`/`Compressor`,
+  `Decompressor`) versus the ~250-line inline implementation it replaces.
+- The compressor's internal (gain-descending) code numbering versus the wire's (length-sorted)
+  numbering is a second permutation the adapter must get exactly right in both directions (symbol
+  table population and code-stream remapping) — a subtle class of bug (wrong output, not a crash)
+  that needs its own dedicated attention in any future change to either ordering.
+
+### Risks to manage
+
+- The branch-free matcher's lack of an input-end bound is a **structural** property, not a bug that
+  was simply fixed once — any future caller of `Matcher.longestMatch`/`Compressor.compress` that
+  operates on a range shorter than 8 bytes from the true end of a real buffer must carry the same
+  `pos + length <= end` bound. This is exactly the kind of thing a future port or refactor could
+  silently drop, since it is not obviously part of the "core algorithm."
+- `LossyPerfectHashTable`'s 2048-slot sizing is a deliberately chosen constant (matching the
+  `spiraldb/fsst` reference), not derived from first principles for this JVM's cache behavior — if a
+  future benchmark shows a different size wins on the actual hardware/workload this project cares
+  about, revisit it deliberately rather than assuming 2048 is universally correct.
+
+## Alternatives considered
+
+- **Reuse an existing Java library.** Searched (GitHub, web) and found none with zero dependencies;
+  the only real Java-ecosystem candidates were the C reference (via JNI, which this project's FFM
+  mandate rules out) and ports in other languages entirely (Rust, Go).
+- **Keep the old inline implementation and just tune constants.** Rejected: the old matching
+  algorithm's O(8)-probes-per-byte structure is the dominant cost, not a constant that tuning could
+  fix — no amount of tuning turns a sequential 8-probe loop into an O(1) branch-free lookup.
+- **`ByteBuffer` instead of `MemorySegment` at the module boundary.** Rejected: reintroduces a dual
+  API (parity tests, two hot-path implementations to keep in sync) for no benefit over
+  `java.lang.foreign`, which is already a zero-dependency JDK-standard API and is what every other
+  module in this codebase uses for the same purpose.
+
+## References
+
+- The paper: Boncz, Neumann, Leis, "FSST: Fast Random Access String Compression", PVLDB 13(11),
+  2020. <https://www.vldb.org/pvldb/vol13/p2649-boncz.pdf>
+- Rust reference: [`spiraldb/fsst`](https://github.com/spiraldb/fsst) on GitHub.
+- The 8-PR sequence (all on `main`):
+  - `2b8db4f2` — module skeleton, `Symbol`, `Decompressor`
+  - `8d778bac` — branch-free matching (`ShortCodeTable` + `LossyPerfectHashTable` + `Matcher`)
+  - `1003a673` — `JavaVsJniFsstBenchmark` baseline (pre-rewrite numbers)
+  - `4158c924` — training (adaptive sampling, min-count pruning, gain boost, final prune) — includes
+    the boundary-overrun fix described in Context
+  - `76e26eb6` — `MemorySegment` hot paths + unconditional-store decode
+  - `1b9714c7` — rewire `writer`/`reader` adapters onto the `fsst` module
+  - `75f58734` — golden test for the paper's Figure 2 worked example
+  - `b218740f` — CHANGELOG before/after benchmark table
diff --git a/adr/ADR.md b/adr/ADR.md
@@ -36,3 +36,4 @@ the decision shipped in (blank = not yet shipped).
 | 0019 | Columnar transducer façade for compute        | Proposed  |          |
 | 0020 | Jazzer fuzz testing infrastructure            | Proposed  |          |
 | 0021 | Cardinality-bounded global dict buffering     | Accepted  |          |
+| 0022 | Extract FSST into a standalone module, ported faithfully from the paper | Accepted |          |
diff --git a/docs/reference.md b/docs/reference.md
@@ -8,6 +8,7 @@ For task-oriented usage see [how-to.md](how-to.md); for design rationale see [ex
 - [Writer API](#writer-api)
 - [Scan API](#scan-api)
 - [Encoding registry](#encoding-registry)
+- [FSST (`io.github.dfa1.vortex.fsst`)](#fsst-iogithubdfa1vortexfsst)
 - [Parquet / CSV import](#parquet--csv-import)
 - [CLI](#cli)
 - [Encoding compatibility](compatibility.md)
@@ -230,6 +231,65 @@ implementations are singletons invoked directly by their `ExtensionId`.
 
 ---
 
+## FSST (`io.github.dfa1.vortex.fsst`)
+
+The `vortex-fsst` module is the standalone FSST (Fast Static Symbol Table) string-compression
+algorithm, usable independently of Vortex. It depends only on the JDK (`java.lang.foreign`), never
+on `core`/`reader`/`writer`; `writer`/`reader` depend on it. The `vortex.fsst` encoding adapter is
+one caller — the module itself knows nothing of the Vortex wire format. See
+[ADR 0022](../adr/0022-fsst-module-extraction.md).
+
+Compress: train a `Compressor` over a corpus, then compress rows against its table. Decompress:
+build a `Decompressor` (from the trained `Compressor`, or from raw table arrays). Hot-path methods
+accept `MemorySegment` (FFM-native, matching the rest of the codebase); `byte[]` overloads exist for
+zero-ceremony standalone use, and both paths produce identical output for the same input.
+
+### `CompressorBuilder`
+
+| Method                  | Notes                                                                       |
+|-------------------------|-----------------------------------------------------------------------------|
+| `new CompressorBuilder()` | Fresh builder using a fixed reproducible default seed                     |
+| `seed(long)`            | Override the training-sample seed; returns `this` for chaining               |
+| `train(byte[][] rows)`  | Runs the bottom-up generation loop, returns a trained `Compressor` (deterministic in rows + seed) |
+
+### `Compressor`
+
+Immutable trained symbol table (up to 255 codes; `0xFF` is the escape). Produced by
+`CompressorBuilder.train(byte[][])`.
+
+| Method                                                    | Notes                                                                 |
+|-----------------------------------------------------------|-----------------------------------------------------------------------|
+| `symbolCount()`                                           | Number of symbols in the table (0–255)                                |
+| `packedSymbol(int code)`                                  | The symbol's bytes packed LSB-first into a `long`                     |
+| `symbolLength(int code)`                                  | The symbol's length in bytes (1–8)                                    |
+| `codesSortedByLength()`                                   | Code permutation, multi-byte length-ascending then all length-1 last  |
+| `toDecompressor()`                                        | A `Decompressor` bound to this table                                  |
+| `compress(byte[] in, int start, int end, byte[] out, long outPos)`             | Greedy longest-match compress; size `out` ≥ `2 * (end - start)`  |
+| `compress(MemorySegment in, long start, long end, MemorySegment out, long outPos)` | FFM-native hot path; same sizing rule                       |
+
+### `Decompressor`
+
+Decodes an FSST code stream against a trained table (parallel arrays indexed by code).
+
+| Method                                                    | Notes                                                                 |
+|-----------------------------------------------------------|-----------------------------------------------------------------------|
+| `static of(long[] packedSymbols, int[] lengths)`          | Bind to a table; arrays are referenced, not copied, and must match in length |
+| `static final int ESCAPE`                                 | The escape code (`0xFF`)                                              |
+| `decompress(byte[] in, int start, int end, byte[] out, long outPos)`           | Decode; size `out` ≥ `8 * (end - start)`                         |
+| `decompress(MemorySegment in, long start, long end, MemorySegment out, long outPos)` | Unconditional-8-byte-store decode; caller MUST allocate `out` with ≥ 7 bytes of trailing slack |
+
+### `Symbol`
+
+`record Symbol(long packedBytes, int length)` — up to 8 bytes packed LSB-first (byte `k` at bit
+`k * 8`), length in 1–8.
+
+| Method                              | Notes                                                       |
+|-------------------------------------|-------------------------------------------------------------|
+| `static of(byte[] data, int offset, int length)` | Pack `length` bytes at `data[offset..]` LSB-first |
+| `byteAt(int i)`                     | The `i`-th byte (0-indexed; byte 0 is the LSB)              |
+
+---
+
 ## Layout registry
 
 ### `LayoutRegistry` (`io.github.dfa1.vortex.reader.layout`)