Skip to content

perf(fsst): close the vortex-jni gap on encode and decode hot paths #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dfa1 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

perf(fsst): close the vortex-jni gap on encode and decode hot paths #300

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

Measured with `JavaVsJniFsstBenchmark` ([1003a673](https://github.com/dfa1/vortex-java/commit/1003a673)), unmodified before and after.

- `vortex.fsst` hot paths close the remaining `vortex-jni` gap on both sides: single 8-byte word loads and a one-cache-line packed hash-slot layout in the compressor, one shared scratch buffer in the encoder (was two heap allocations per row), and prefix-sum output offsets plus 256-row batched decompression in the decoder (was a per-row offset `switch` and two broadcast modulos per row). ([#300](https://github.com/dfa1/vortex-java/pull/300))

| Benchmark | Before | After | `vortex-java` vs `vortex-jni` |
|---|---|---|---|
| `javaFsstEncode` | 1.848 ops/s | 3.059 ± 0.430 ops/s | 1.6x slower → parity (1.07x, overlapping error) |
| `javaFsstDecode` | 27.137 ops/s | 32.924 ± 0.824 ops/s | 1.3x slower → 1.16x faster |

Measured with `JavaVsJniFsstBenchmark` `-f 3` (jni: encode 2.854 ± 0.289, decode 28.353 ± 5.283 ops/s on the same machine and run).

### Fixed

- `ParquetImporter` detects duplicate column names in the source Parquet schema and throws a clear message naming the duplicate(s) and the source file, instead of a confusing `VortexWriter` internal-invariant error several frames removed from the actual cause. ([f2c05e57](https://github.com/dfa1/vortex-java/commit/f2c05e57))
Expand Down
11 changes: 6 additions & 5 deletions TODO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@

### FSST follow-ups

Out of scope for the #287 rewrite (which closed most of the `vortex-jni` gap with a scalar,
branch-free algorithm — see [ADR-0022](adr/0022-fsst-module-extraction.md)):
Out of scope for the #287 rewrite (which, together with the follow-up hot-path pass, closed the
`vortex-jni` gap with a scalar, branch-free algorithm — encode at parity, decode ~1.16x faster on
`JavaVsJniFsstBenchmark` — see [ADR-0022](adr/0022-fsst-module-extraction.md)):

- [ ] **AVX512/SIMD compression kernel** — the paper measures a SIMD kernel as the fastest known
string compressor at that tier, but this project's scalar rewrite already narrowed the encode gap
to `vortex-jni` from 36x to 1.6x. Needs a Vector-API/JDK-incubator decision and its own ADR (cf.
[ADR-0005](adr/0005-vector-api-adoption.md)).
string compressor at that tier. The scalar path now matches `vortex-jni` (itself scalar), so a
SIMD kernel is about beating the reference, not catching it. Needs a Vector-API/JDK-incubator
decision and its own ADR (cf. [ADR-0005](adr/0005-vector-api-adoption.md)).
- [ ] **True per-row lazy/random-access decompression** exploiting FSST's headline random-access
property — today `FsstEncodingDecoder.decode()` eagerly materializes the whole column up front
regardless of what is queried. Connects to [ADR-0010](adr/0010-lazy-decode.md) (Lazy decode) but
Expand Down
57 changes: 52 additions & 5 deletions fsst/src/main/java/io/github/dfa1/vortex/fsst/Compressor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

import java.lang.foreign.MemorySegment;
import java.lang.foreign.ValueLayout;
import java.lang.invoke.MethodHandles;
import java.lang.invoke.VarHandle;
import java.nio.ByteOrder;
import java.util.ArrayList;
import java.util.List;

Expand All @@ -23,7 +26,13 @@ public final class Compressor {
/// `core`'s `VortexFormat.LE_LONG` so this module keeps its zero dependency on `core`; the FSST
/// wire format packs symbol bytes LSB-first, so all `MemorySegment` reads/writes are little-endian.
static final ValueLayout.OfLong LE_LONG =
ValueLayout.JAVA_LONG_UNALIGNED.withOrder(java.nio.ByteOrder.LITTLE_ENDIAN);
ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);

/// Little-endian unaligned `long` view over a `byte[]`, the JIT-intrinsified way to load an
/// 8-byte input word in one instruction on the `byte[]` compress fast path (the byte-by-byte
/// [#loadWord(byte[], int, int)] assembly is 8 loads + shifts and is kept for the tail only).
private static final VarHandle LONG_LE_BYTES =
MethodHandles.byteArrayViewVarHandle(long[].class, ByteOrder.LITTLE_ENDIAN);

private final List<Symbol> symbolsByGainDescending;
private final Matcher matcher;
Expand Down Expand Up @@ -142,8 +151,27 @@ public Decompressor toDecompressor() {
public long compress(byte[] input, int start, int end, byte[] out, long outPos) {
int pos = start;
int outIndex = (int) outPos;
// Fast path: while 8 real input bytes remain, load the word in one intrinsified read. No
// over-long-match guard is needed here — a match is at most 8 bytes, so it can never reach
// past end while pos <= end - 8.
int fastEnd = end - 8;
while (pos <= fastEnd) {
long word = (long) LONG_LE_BYTES.get(input, pos);
int packedMatch = matcher.longestMatch(word);
int length = Matcher.lengthOf(packedMatch);
if (length > 0) {
out[outIndex++] = (byte) Matcher.codeOf(packedMatch);
pos += length;
} else {
// The escaped literal is the word's low byte — already loaded, no re-read.
out[outIndex++] = (byte) ESCAPE;
out[outIndex++] = (byte) word;
pos++;
}
}
while (pos < end) {
int packedMatch = matcher.longestMatch(loadWord(input, pos, end));
long word = loadWord(input, pos, end);
int packedMatch = matcher.longestMatch(word);
int length = Matcher.lengthOf(packedMatch);
// Reject an over-long match: loadWord zero-pads bytes past end, and the branch-free
// Matcher has no notion of end, so a symbol whose trailing bytes are zero can spuriously
Expand All @@ -156,7 +184,7 @@ public long compress(byte[] input, int start, int end, byte[] out, long outPos)
pos += length;
} else {
out[outIndex++] = (byte) ESCAPE;
out[outIndex++] = input[pos];
out[outIndex++] = (byte) word;
pos++;
}
}
Expand Down Expand Up @@ -187,8 +215,27 @@ public long compress(byte[] input, int start, int end, byte[] out, long outPos)
public long compress(MemorySegment input, long start, long end, MemorySegment out, long outPos) {
long pos = start;
long outIndex = outPos;
// Fast path: while 8 readable bytes remain (bounded by both end and the segment's own
// size — the segment may end exactly at end with no trailing slack), load the word in one
// unaligned read. No over-long-match guard is needed here — a match is at most 8 bytes.
long fastEnd = Math.min(end, input.byteSize()) - 8;
while (pos <= fastEnd) {
long word = input.get(LE_LONG, pos);
int packedMatch = matcher.longestMatch(word);
int length = Matcher.lengthOf(packedMatch);
if (length > 0) {
out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) Matcher.codeOf(packedMatch));
pos += length;
} else {
// The escaped literal is the word's low byte — already loaded, no re-read.
out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) ESCAPE);
out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) word);
pos++;
}
}
while (pos < end) {
int packedMatch = matcher.longestMatch(loadWord(input, pos, end));
long word = loadWord(input, pos, end);
int packedMatch = matcher.longestMatch(word);
int length = Matcher.lengthOf(packedMatch);
// Same boundary guard as the byte[] overload: reject a match that reaches past the real
// input end, since loadWord zero-pads and the branch-free Matcher has no notion of end.
Expand All @@ -197,7 +244,7 @@ public long compress(MemorySegment input, long start, long end, MemorySegment ou
pos += length;
} else {
out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) ESCAPE);
out.set(ValueLayout.JAVA_BYTE, outIndex++, input.get(ValueLayout.JAVA_BYTE, pos));
out.set(ValueLayout.JAVA_BYTE, outIndex++, (byte) word);
pos++;
}
}
Expand Down
93 changes: 34 additions & 59 deletions fsst/src/main/java/io/github/dfa1/vortex/fsst/LossyPerfectHashTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,30 +35,21 @@ final class LossyPerfectHashTable {
/// Low three bytes of the input word — the only bytes the hash keys on.
private static final long PREFIX_MASK = 0x00FF_FFFFL;

/// Per-slot packed symbol bytes, LSB-first ([Symbol] convention). Meaningful only where the
/// corresponding [#occupied] entry is set.
private final long[] packedBytes;

/// Per-slot mask `~0L >>> ignoredBits` (`ignoredBits = 64 - 8 * length`) applied to an input
/// word before comparing, clearing the high bytes past the candidate's length.
private final long[] keepMask;

/// Per-slot symbol length in bytes, 3-8.
private final int[] lengths;

/// Per-slot code (the symbol's list index), meaningful only where [#occupied] is set.
private final int[] codes;

/// Whether each slot holds a candidate.
private final boolean[] occupied;

private LossyPerfectHashTable(long[] packedBytes, long[] keepMask, int[] lengths, int[] codes,
boolean[] occupied) {
this.packedBytes = packedBytes;
this.keepMask = keepMask;
this.lengths = lengths;
this.codes = codes;
this.occupied = occupied;
/// Slot table with two adjacent longs per slot (the FSST paper's C layout): `slots[2 * s]` is
/// the candidate's packed symbol bytes (LSB-first, [Symbol] convention) and `slots[2 * s + 1]`
/// is its metadata `ignoredBits << 16 | code << 8 | length` (`ignoredBits = 64 - 8 * length`).
/// A lookup derives the keep-mask from the ignored-bits with one shift instead of loading a
/// separate mask array, so the whole 16-byte slot lands in a single cache line — one memory
/// touch per lookup instead of three parallel-array reads across three lines.
///
/// Empty slots are all-zero: metadata 0 makes the keep-mask `~0L` and the symbol 0, so an
/// input word of exactly 0 can "hit" an empty slot — harmlessly, because the returned low 16
/// bits (`code << 8 | length`) are then 0, which is precisely the "no match" answer. No
/// occupancy flag or sentinel is needed.
private final long[] slots;

private LossyPerfectHashTable(long[] slots) {
this.slots = slots;
}

/// Builds the table from the trained symbols in descending-gain order, keeping only those of
Expand All @@ -76,43 +67,41 @@ private LossyPerfectHashTable(long[] packedBytes, long[] keepMask, int[] lengths
/// @param symbolsByGainDescending the trained symbols, code = list index, gain-descending
/// @return a hash table resolving 3-8 byte matches with first-writer-wins on collision
static LossyPerfectHashTable of(List<Symbol> symbolsByGainDescending) {
long[] packedBytes = new long[SLOTS];
long[] keepMask = new long[SLOTS];
int[] lengths = new int[SLOTS];
int[] codes = new int[SLOTS];
boolean[] occupied = new boolean[SLOTS];
long[] slots = new long[2 * SLOTS];
for (int code = 0; code < symbolsByGainDescending.size(); code++) {
Symbol symbol = symbolsByGainDescending.get(code);
if (symbol.length() < 3) {
continue; // Length 1-2 belongs to ShortCodeTable.
}
int slot = slotFor(symbol.packedBytes());
if (occupied[slot]) {
if (slots[2 * slot + 1] != 0) {
continue; // First writer (higher gain) wins; skip the collision.
}
occupied[slot] = true;
packedBytes[slot] = symbol.packedBytes();
keepMask[slot] = keepMaskFor(symbol.length());
lengths[slot] = symbol.length();
codes[slot] = code;
long ignoredBits = 64L - 8 * symbol.length();
slots[2 * slot] = symbol.packedBytes();
slots[2 * slot + 1] = ignoredBits << 16 | (long) code << 8 | symbol.length();
}
return new LossyPerfectHashTable(packedBytes, keepMask, lengths, codes, occupied);
return new LossyPerfectHashTable(slots);
}

/// Looks up `word` and reports whether a stored 3-8 byte symbol really matches it.
/// Looks up `word` and returns the matched symbol as `code << 8 | length`, or 0 when no stored
/// 3-8 byte symbol matches.
///
/// The input word's first three bytes select one slot; the candidate there matches only if the
/// masked compare passes (`(word & keepMask) == candidate.packedBytes`), which rules out both
/// hash collisions with unrelated bytes and empty slots. On a miss the returned [HashMatch] has
/// `hit == false` and the caller falls back to [ShortCodeTable].
/// masked compare passes (`(word & (~0L >>> ignoredBits)) == candidate.packedBytes`), which
/// rules out hash collisions with unrelated bytes. An empty slot can only "hit" an all-zero
/// input word, and then still answers 0 (its metadata is 0), which is the no-match result. A
/// stored symbol's length is 3-8, so a real hit is never 0 and the caller can test the result
/// directly. On a miss the caller falls back to [ShortCodeTable].
///
/// @param word an 8-byte little-endian input word starting at the current match position, with
/// any bytes past the remaining input already zero-padded by the caller
/// @return the match result: `hit`, and when hit the matched `code` and `length`
HashMatch lookup(long word) {
int slot = slotFor(word);
boolean hit = occupied[slot] && (word & keepMask[slot]) == packedBytes[slot];
return new HashMatch(hit, codes[slot], lengths[slot]);
/// @return the match as `code << 8 | length`, or 0 when there is no match
int lookup(long word) {
int slot = slotFor(word) << 1;
long symbol = slots[slot];
long meta = slots[slot + 1];
return (word & (~0L >>> (int) (meta >>> 16))) == symbol ? (int) (meta & 0xFFFF) : 0;
}

/// Computes the slot index a word hashes to, keyed on its first three bytes. Package-visible so
Expand All @@ -126,18 +115,4 @@ static int slotFor(long word) {
return (int) (mixed >>> 32) & SLOT_MASK;
}

private static long keepMaskFor(int length) {
int ignoredBits = 64 - 8 * length;
return ~0L >>> ignoredBits;
}

/// Result of a [LossyPerfectHashTable#lookup(long)]: whether a 3-8 byte symbol matched and, if
/// so, its code and length. When `hit` is false, `code` and `length` are unspecified and the
/// caller must ignore them.
///
/// @param hit whether a stored symbol really matched the input word
/// @param code the matched symbol code, valid only when `hit`
/// @param length the matched symbol length in bytes (3-8), valid only when `hit`
record HashMatch(boolean hit, int code, int length) {
}
}
12 changes: 7 additions & 5 deletions fsst/src/main/java/io/github/dfa1/vortex/fsst/Matcher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,13 @@ public static Matcher of(List<Symbol> symbolsByGainDescending) {
/// any bytes past the remaining input already zero-padded by the caller
/// @return the longest match as `code << 8 | length`; length 0 signals "no match, escape"
public int longestMatch(long word) {
LossyPerfectHashTable.HashMatch hashMatch = hashTable.lookup(word);
if (hashMatch.hit()) {
return hashMatch.code() << 8 | hashMatch.length();
}
return shortCodes.codeFor(word) << 8 | shortCodes.lengthFor(word);
// Both tables are read unconditionally so the select below has no side to skip — the JIT
// can lower it to a conditional move instead of a data-dependent branch, which matters
// because hash hit/miss alternates unpredictably across input positions. The extra
// short-code read on a hash hit is one L1 load into a 256 KB table.
int hashMatch = hashTable.lookup(word);
int shortMatch = shortCodes.packedFor(word);
return hashMatch != 0 ? hashMatch : shortMatch;
}

/// Extracts the code from a packed [#longestMatch(long)] result.
Expand Down
30 changes: 23 additions & 7 deletions fsst/src/main/java/io/github/dfa1/vortex/fsst/ShortCodeTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package io.github.dfa1.vortex.fsst;

import java.util.Arrays;
import java.util.List;

/// Direct-indexed table resolving the shortest FSST matches — length 0 (no match), 1, or 2 —
Expand All @@ -25,7 +26,12 @@ final class ShortCodeTable {
/// codes are `0..254` (`0xFF` is the escape), so `-1` can never collide with a real code.
static final int NO_CODE = -1;

/// Packed `code << 8 | length` per 16-bit key. A zero length marks "no match".
/// Packed no-match value stored in unpopulated slots: `NO_CODE << 8 | 0`. Storing the sentinel
/// pre-packed lets [#packedFor(long)] return the slot verbatim — one array read, no branch —
/// and an arithmetic `>> 8` recovers [#NO_CODE] while `& 0xFF` recovers length 0.
private static final int NO_MATCH = NO_CODE << 8;

/// Packed `code << 8 | length` per 16-bit key. A zero length marks "no match" ([#NO_MATCH]).
private final int[] slots;

private ShortCodeTable(int[] slots) {
Expand All @@ -46,6 +52,7 @@ private ShortCodeTable(int[] slots) {
/// @return a table resolving 0/1/2-byte matches for any two-byte input prefix
static ShortCodeTable of(List<Symbol> symbolsByGainDescending) {
int[] slots = new int[SLOTS];
Arrays.fill(slots, NO_MATCH);
for (int code = 0; code < symbolsByGainDescending.size(); code++) {
Symbol symbol = symbolsByGainDescending.get(code);
if (symbol.length() == 1) {
Expand All @@ -69,23 +76,32 @@ static ShortCodeTable of(List<Symbol> symbolsByGainDescending) {
return new ShortCodeTable(slots);
}

/// Returns the code matched by the low two bytes of `word`, or [#NO_CODE] if none. One array
/// read, no branching.
/// Returns the match for the low two bytes of `word` as `code << 8 | length`, or
/// `NO_CODE << 8` (length 0) when there is no length-1 or length-2 match. One array read, no
/// branching — the slot value is returned verbatim, which is what makes this the hot path's
/// fallback of choice.
///
/// @param word an input word; only its low 16 bits (first two input bytes) are consulted
/// @return the match as `code << 8 | length`; length 0 (and code [#NO_CODE]) means no match
int packedFor(long word) {
return slots[(int) (word & 0xFFFF)];
}

/// Returns the code matched by the low two bytes of `word`, or [#NO_CODE] if none.
///
/// @param word an input word; only its low 16 bits (first two input bytes) are consulted
/// @return the matched symbol code, or [#NO_CODE] when there is no length-1 or length-2 match
int codeFor(long word) {
int packed = slots[(int) (word & 0xFFFF)];
return packed == 0 ? NO_CODE : packed >>> 8;
return packedFor(word) >> 8;
}

/// Returns the length of the symbol matched by the low two bytes of `word`: 2, 1, or 0 for no
/// match. One array read, no branching.
/// match.
///
/// @param word an input word; only its low 16 bits (first two input bytes) are consulted
/// @return the matched symbol length in bytes, or 0 when there is no match
int lengthFor(long word) {
return length(slots[(int) (word & 0xFFFF)]);
return length(packedFor(word));
}

private static int length(int packed) {
Expand Down
Loading