diff --git a/CLAUDE.md b/CLAUDE.md index 566f3148..164d3826 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -22,6 +22,10 @@ Benchmark classes follow this: `JavaVsJni{Read,Write,Filter}Benchmark`, ## Module structure ``` +fsst — io.github.dfa1.vortex.fsst: standalone FSST (Fast Static Symbol Table) string + compression algorithm — Symbol, Compressor/CompressorBuilder, Decompressor/Matcher. + Zero dependency on core/reader/writer/FFM-wire-concerns beyond the JDK's own + java.lang.foreign; writer/reader depend on it, not the other way around. core — everything lives under `io.github.dfa1.vortex.core.*`: core.model DType, PType, TimeUnit, EncodingId, LayoutId, ColumnName, ExtensionId, TimeDtype, TimestampDtype core.io IoBounds, PTypeIO, VortexFormat diff --git a/TODO.md b/TODO.md index c694c430..58ca09b4 100644 --- a/TODO.md +++ b/TODO.md @@ -13,6 +13,24 @@ - [ ] Run performance tests on other machines (I have access only to Apple M5) - [ ] **Vector API adoption** — see [ADR-0005](adr/0005-vector-api-adoption.md). +### FSST follow-ups + +Out of scope for the #287 rewrite (which closed most of the `vortex-jni` gap with a scalar, +branch-free algorithm — see [ADR-0022](adr/0022-fsst-module-extraction.md)): + +- [ ] **AVX512/SIMD compression kernel** — the paper measures a SIMD kernel as the fastest known + string compressor at that tier, but this project's scalar rewrite already narrowed the encode gap + to `vortex-jni` from 36x to 1.6x. Needs a Vector-API/JDK-incubator decision and its own ADR (cf. + [ADR-0005](adr/0005-vector-api-adoption.md)). +- [ ] **True per-row lazy/random-access decompression** exploiting FSST's headline random-access + property — today `FsstEncodingDecoder.decode()` eagerly materializes the whole column up front + regardless of what is queried. Connects to [ADR-0010](adr/0010-lazy-decode.md) (Lazy decode) but + is a separate initiative. +- [ ] **OptFSST** (2026 arXiv follow-up: DP-based training instead of greedy, ~4x slower training + for 7–17% better compression) — a documented future option, not adopted, since it moves off the + classic greedy-FSST speed/compression tradeoff this rewrite targets (matching what `vortex-jni` + itself uses). + ## Security See [CLAUDE.md §Security contract](CLAUDE.md) for the invariant. Each entry below is either a diff --git a/adr/0022-fsst-module-extraction.md b/adr/0022-fsst-module-extraction.md new file mode 100644 index 00000000..55909a61 --- /dev/null +++ b/adr/0022-fsst-module-extraction.md @@ -0,0 +1,145 @@ +# ADR 0022: Extract FSST into a standalone module, ported faithfully from the paper + +- **Status:** Accepted +- **Date:** 2026-07-20 +- **Deciders:** project maintainer +- **Supersedes:** — +- **Superseded by:** — +- **Related:** [ADR 0005 — Vector API adoption](0005-vector-api-adoption.md), [ADR 0010 — Lazy + decode](0010-lazy-decode.md), [ADR 0017 — In-house FlatBuffers codegen](0017-in-house-flatbuffers-codegen.md) + +## Context + +`writer/FsstEncodingEncoder` and `reader/FsstEncodingDecoder` held a first-pass, simplified FSST +(Fast Static Symbol Table) implementation. Comparing it against `vortex-jni` (issue #287) surfaced +real gaps against both the original paper (Boncz/Neumann/Leis, "FSST: Fast Random Access String +Compression", PVLDB 13(11), 2020) and the Rust reference (`spiraldb/fsst`, the crate the Rust +`vortex-fsst` wire adapter is built on): + +- **Matching.** The old `longestMatch` probed symbol lengths 8 down to 1 in a per-position loop — + up to eight sequential open-addressing hash lookups per input byte, a variable-length loop body + that blocks JIT auto-vectorization (the exact shape CLAUDE.md's hot-loop rule forbids). The + paper's Algorithm 4 ("lossy perfect hashing", §5.1) resolves any match in O(1): a direct + 65536-entry array for 0/1/2-byte symbols, and a small hash table for 3-8 byte symbols, combined + branch-free. +- **Training.** The old bottom-up loop always compress-counted the *full* training sample on every + one of its five generations, ranked candidates by plain `count * length`, and never pruned + low-frequency noise. The real reference (`spiraldb/fsst`) additionally uses a growing + per-generation sample fraction (~6% → 100%), a per-generation min-count floor, an 8x gain boost + for single-byte candidates (to suppress escapes), and a final cost-based prune pass. +- **Decode.** The old decoder copied a matched symbol's bytes one at a time in a per-byte loop + instead of the paper's Algorithm 1 trick — one unconditional 8-byte store, advancing the output + cursor by only the symbol's true length. + +A zero-dependency Java FSST implementation was searched for and does not exist (only the C +reference `cwida/fsst`, the Rust `spiraldb/fsst`, and a Go port `axiomhq/fsst`). Per that search's +own conclusion, the fix isolates FSST into its own module — mirroring how the Rust reference itself +splits a pure algorithm crate (`spiraldb/fsst`, zero dependencies, no Vortex awareness) from +vortex-rust's thin wire adapter (`vortex-fsst`) — so the compression algorithm can be iterated on +and benchmarked without the Vortex wire-format "shell" in the way. + +During the rewrite's own review, a real data-corruption bug was found and fixed: the new +branch-free matcher has no notion of "end of input" (unlike the old bounded loop), so a trained +symbol whose trailing bytes happen to be zero could spuriously match zero-padded bytes past a short +input's true end, silently appending garbage bytes to the decoded output. Confirmed with a concrete +repro (a 3-byte input round-tripping to 8 bytes) before it reached a merged PR. Fixed by an explicit +`pos + length <= end` guard at both call sites that feed the matcher a possibly-short remaining +range (`Compressor.compress` and the training loop's compress-count step) — the paper's own +guidance for a branch-free scalar kernel (§5.2) names exactly this bound as the alternative to a +terminator byte (which is an AVX512-batch-kernel-specific optimization, out of scope here). + +## Decision + +Extract FSST into a new top-level Maven module, `fsst` (artifact `vortex-fsst`), with zero +dependency on `core`/`reader`/`writer`: + +1. **Port the paper faithfully**, plus the real reference's engineering refinements: branch-free + O(1) matching (`ShortCodeTable` for 0/1/2-byte symbols, a 2048-slot lossy perfect hash table for + 3-8 byte symbols — matching `spiraldb/fsst`'s L1D-cache-line-split sizing over the paper's + literal 4096, since that Rust crate is the actual `vortex-jni` comparison target); adaptive + byte-size-bounded sampling with a growing per-generation fraction, min-count pruning, the + single-byte gain boost, and the final cost-based prune during training; the unconditional-8-byte- + store decode trick. +2. **Memory boundary: `MemorySegment`, not `ByteBuffer`.** `java.lang.foreign` is a standard JDK + module, not a third-party dependency — using it satisfies "zero deps" exactly as well as + `ByteBuffer` would, while staying consistent with every other module in this codebase (CLAUDE.md: + "Uses FFM (`MemorySegment`/`Arena`) — never JNI or `sun.misc.Unsafe`") and avoiding a dual-API + translation layer with its own parity-testing burden. The module's hot-path `compress`/ + `decompress` methods take `MemorySegment` directly; plain `byte[]` overloads exist alongside them + for standalone use with zero FFM ceremony. +3. **`writer`/`reader` become thin wire adapters.** `FsstEncodingEncoder`/`FsstEncodingDecoder` keep + all Vortex-specific plumbing (UTF-8 conversion, `ProtoFSSTMetadata`, `EncodeNode`/`EncodeResult` + assembly, `Arena` allocation) and delegate the algorithm itself to `CompressorBuilder`/ + `Compressor`/`Decompressor`. The wire format (`vortex.fsst`) is unchanged — this is a pure + algorithm/performance rewrite, not a wire-format change. The one adapter-side subtlety: the + module numbers codes gain-descending internally (load-bearing for the hash table's + first-writer-wins collision rule), while the wire format requires length-sorted order + (`Compressor#codesSortedByLength()`), so the writer remaps every code the compressor emits + through the wire permutation before writing it out. + +## Consequences + +### Positive + +- Measured, real speedup (`JavaVsJniFsstBenchmark`, unmodified before and after — see the + `[Unreleased]` CHANGELOG entry for the full table): `javaFsstEncode` 0.085 → 1.848 ops/s (~21.7x + faster, closing the gap to `vortex-jni` from 36x slower to 1.6x slower); `javaFsstDecode` 5.243 → + 27.137 ops/s (~5.2x faster, gap 6.5x → 1.3x). +- The `fsst` module is independently testable and benchmarkable with zero Vortex file-format + scaffolding in the loop (`FsstEncodingEncoderTest`'s wire-format tests are unaffected; new + `fsst`-module tests exercise the algorithm directly) — the issue's own stated motivation for the + extraction. +- A golden test (`PaperFigure2Test`) pins the paper's own worked example against a hand-built + (not trained) symbol table — an external, human-verifiable fixed point independent of this + project's training heuristics. + +### Negative + +- Real rework: five new PRs' worth of module surface (`Symbol`, `Matcher`/`ShortCodeTable`/ + `LossyPerfectHashTable`, `Sample`/`TrainingGeneration`/`CompressorBuilder`/`Compressor`, + `Decompressor`) versus the ~250-line inline implementation it replaces. +- The compressor's internal (gain-descending) code numbering versus the wire's (length-sorted) + numbering is a second permutation the adapter must get exactly right in both directions (symbol + table population and code-stream remapping) — a subtle class of bug (wrong output, not a crash) + that needs its own dedicated attention in any future change to either ordering. + +### Risks to manage + +- The branch-free matcher's lack of an input-end bound is a **structural** property, not a bug that + was simply fixed once — any future caller of `Matcher.longestMatch`/`Compressor.compress` that + operates on a range shorter than 8 bytes from the true end of a real buffer must carry the same + `pos + length <= end` bound. This is exactly the kind of thing a future port or refactor could + silently drop, since it is not obviously part of the "core algorithm." +- `LossyPerfectHashTable`'s 2048-slot sizing is a deliberately chosen constant (matching the + `spiraldb/fsst` reference), not derived from first principles for this JVM's cache behavior — if a + future benchmark shows a different size wins on the actual hardware/workload this project cares + about, revisit it deliberately rather than assuming 2048 is universally correct. + +## Alternatives considered + +- **Reuse an existing Java library.** Searched (GitHub, web) and found none with zero dependencies; + the only real Java-ecosystem candidates were the C reference (via JNI, which this project's FFM + mandate rules out) and ports in other languages entirely (Rust, Go). +- **Keep the old inline implementation and just tune constants.** Rejected: the old matching + algorithm's O(8)-probes-per-byte structure is the dominant cost, not a constant that tuning could + fix — no amount of tuning turns a sequential 8-probe loop into an O(1) branch-free lookup. +- **`ByteBuffer` instead of `MemorySegment` at the module boundary.** Rejected: reintroduces a dual + API (parity tests, two hot-path implementations to keep in sync) for no benefit over + `java.lang.foreign`, which is already a zero-dependency JDK-standard API and is what every other + module in this codebase uses for the same purpose. + +## References + +- The paper: Boncz, Neumann, Leis, "FSST: Fast Random Access String Compression", PVLDB 13(11), + 2020. +- Rust reference: [`spiraldb/fsst`](https://github.com/spiraldb/fsst) on GitHub. +- The 8-PR sequence (all on `main`): + - `2b8db4f2` — module skeleton, `Symbol`, `Decompressor` + - `8d778bac` — branch-free matching (`ShortCodeTable` + `LossyPerfectHashTable` + `Matcher`) + - `1003a673` — `JavaVsJniFsstBenchmark` baseline (pre-rewrite numbers) + - `4158c924` — training (adaptive sampling, min-count pruning, gain boost, final prune) — includes + the boundary-overrun fix described in Context + - `76e26eb6` — `MemorySegment` hot paths + unconditional-store decode + - `1b9714c7` — rewire `writer`/`reader` adapters onto the `fsst` module + - `75f58734` — golden test for the paper's Figure 2 worked example + - `b218740f` — CHANGELOG before/after benchmark table diff --git a/adr/ADR.md b/adr/ADR.md index fecdd1ae..dbc99f7c 100644 --- a/adr/ADR.md +++ b/adr/ADR.md @@ -36,3 +36,4 @@ the decision shipped in (blank = not yet shipped). | 0019 | Columnar transducer façade for compute | Proposed | | | 0020 | Jazzer fuzz testing infrastructure | Proposed | | | 0021 | Cardinality-bounded global dict buffering | Accepted | | +| 0022 | Extract FSST into a standalone module, ported faithfully from the paper | Accepted | | diff --git a/docs/reference.md b/docs/reference.md index 5a50e6c8..317d65b3 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -8,6 +8,7 @@ For task-oriented usage see [how-to.md](how-to.md); for design rationale see [ex - [Writer API](#writer-api) - [Scan API](#scan-api) - [Encoding registry](#encoding-registry) +- [FSST (`io.github.dfa1.vortex.fsst`)](#fsst-iogithubdfa1vortexfsst) - [Parquet / CSV import](#parquet--csv-import) - [CLI](#cli) - [Encoding compatibility](compatibility.md) @@ -230,6 +231,65 @@ implementations are singletons invoked directly by their `ExtensionId`. --- +## FSST (`io.github.dfa1.vortex.fsst`) + +The `vortex-fsst` module is the standalone FSST (Fast Static Symbol Table) string-compression +algorithm, usable independently of Vortex. It depends only on the JDK (`java.lang.foreign`), never +on `core`/`reader`/`writer`; `writer`/`reader` depend on it. The `vortex.fsst` encoding adapter is +one caller — the module itself knows nothing of the Vortex wire format. See +[ADR 0022](../adr/0022-fsst-module-extraction.md). + +Compress: train a `Compressor` over a corpus, then compress rows against its table. Decompress: +build a `Decompressor` (from the trained `Compressor`, or from raw table arrays). Hot-path methods +accept `MemorySegment` (FFM-native, matching the rest of the codebase); `byte[]` overloads exist for +zero-ceremony standalone use, and both paths produce identical output for the same input. + +### `CompressorBuilder` + +| Method | Notes | +|-------------------------|-----------------------------------------------------------------------------| +| `new CompressorBuilder()` | Fresh builder using a fixed reproducible default seed | +| `seed(long)` | Override the training-sample seed; returns `this` for chaining | +| `train(byte[][] rows)` | Runs the bottom-up generation loop, returns a trained `Compressor` (deterministic in rows + seed) | + +### `Compressor` + +Immutable trained symbol table (up to 255 codes; `0xFF` is the escape). Produced by +`CompressorBuilder.train(byte[][])`. + +| Method | Notes | +|-----------------------------------------------------------|-----------------------------------------------------------------------| +| `symbolCount()` | Number of symbols in the table (0–255) | +| `packedSymbol(int code)` | The symbol's bytes packed LSB-first into a `long` | +| `symbolLength(int code)` | The symbol's length in bytes (1–8) | +| `codesSortedByLength()` | Code permutation, multi-byte length-ascending then all length-1 last | +| `toDecompressor()` | A `Decompressor` bound to this table | +| `compress(byte[] in, int start, int end, byte[] out, long outPos)` | Greedy longest-match compress; size `out` ≥ `2 * (end - start)` | +| `compress(MemorySegment in, long start, long end, MemorySegment out, long outPos)` | FFM-native hot path; same sizing rule | + +### `Decompressor` + +Decodes an FSST code stream against a trained table (parallel arrays indexed by code). + +| Method | Notes | +|-----------------------------------------------------------|-----------------------------------------------------------------------| +| `static of(long[] packedSymbols, int[] lengths)` | Bind to a table; arrays are referenced, not copied, and must match in length | +| `static final int ESCAPE` | The escape code (`0xFF`) | +| `decompress(byte[] in, int start, int end, byte[] out, long outPos)` | Decode; size `out` ≥ `8 * (end - start)` | +| `decompress(MemorySegment in, long start, long end, MemorySegment out, long outPos)` | Unconditional-8-byte-store decode; caller MUST allocate `out` with ≥ 7 bytes of trailing slack | + +### `Symbol` + +`record Symbol(long packedBytes, int length)` — up to 8 bytes packed LSB-first (byte `k` at bit +`k * 8`), length in 1–8. + +| Method | Notes | +|-------------------------------------|-------------------------------------------------------------| +| `static of(byte[] data, int offset, int length)` | Pack `length` bytes at `data[offset..]` LSB-first | +| `byteAt(int i)` | The `i`-th byte (0-indexed; byte 0 is the LSB) | + +--- + ## Layout registry ### `LayoutRegistry` (`io.github.dfa1.vortex.reader.layout`)