From 820859cb90bdf24b52a6582e3fdf553096b0ceb1 Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Tue, 21 Jul 2026 22:41:05 +0200 Subject: [PATCH] docs: zero-copy mmap-vs-jni story, JMH-backed Answers whether mapping a large file into one MemorySegment and streaming it through ZstdCompressStream beats zstd-jni's classic ZstdOutputStream-over-InputStream path - for capability, throughput, allocation, and code complexity. Capability (unconditional): FileChannel.map(mode, pos, size, Arena) is long-indexed and maps a file of any size in one call. zstd-jni's zero-copy surface is ByteBuffer-based and can't map past Integer.MAX_VALUE at all. Proven by LargeMemoryMappedFileTest, a disabled-by-default integration test gated behind -Dzstd.test.large=true (writes several GiB to disk). Throughput and allocation (JMH-backed): adds benchmark/.../LargeFileBenchmark, replacing the ad hoc System.nanoTime scratch harnesses that produced earlier, implausible numbers for this comparison. 4 variants (mmap, mmap+WILLNEED, this library's ZstdOutputStream, zstd-jni's ZstdOutputStream) x 5 sizes (4 MiB - 10 GiB), plus -prof gc allocation data. Along the way, fixed a real bug the switch to JMH surfaced: the mmap path bounced compressed output through a heap byte[] before handing it to the sink, defeating the zero-copy point - now uses a direct ByteBuffer view over the native dst segment + a WritableByteChannel sink. Also dropped the zstdJniTransferTo variant (transferTo's internal buffer size isn't configurable, so it couldn't be made to match the other variants' 8 MiB CHUNK) and the redundant BufferedInputStream wrap on the other stream variants. Corrected numbers: mmap (especially advised) wins clearly up to 4 GiB (~20-57% faster); at 10 GiB it's a statistical tie with zstd-jni's stream path, not the ~24% mmap win the ad hoc harness had suggested. mmap is allocation-free at every size; both streaming paths allocate a constant ~8.1 MiB/op (their fixed read buffer), which is negligible at 10 GiB but causes hundreds of GCs at 4 MiB where thousands of ops run per JMH measurement window. Ergonomics (new): this library's own ZstdOutputStream is as easy to use as zstd-jni's, but is not zero-copy (still copies byte[] to native per chunk) and is not faster. The genuinely zero-copy MemorySegment/ZstdCompressStream path is real but ~3x more code, and its throughput edge shrinks to a tie right at the scale where the "no 2 GiB cap" capability claim starts to matter most. Recommends the low-level path only when the MemorySegment is already in hand for other reasons (e.g. a memory-mapped reader), not for mmap-from-scratch-to-compress. docs/zero-copy.md gains a full time+allocation reference table across all sizes with machine specs and a reproduce command. Also deletes the two throwaway scratch timing harnesses this benchmark supersedes (integration-tests/.../scratch/ ZstdWithMadviseTimingMain, IoOnlyTimingMain) - both untracked, both marked "deleted with its throwaway branch" in their own header comments. Co-Authored-By: Claude Sonnet 5 --- benchmark/README.md | 25 +- .../dfa1/zstd/bench/LargeFileBenchmark.java | 292 ++++++++++++++++++ docs/zero-copy.md | 247 ++++++++++++++- .../zstd/it/LargeMemoryMappedFileTest.java | 263 ++++++++++++++++ 4 files changed, 824 insertions(+), 3 deletions(-) create mode 100644 benchmark/src/main/java/io/github/dfa1/zstd/bench/LargeFileBenchmark.java create mode 100644 integration-tests/src/test/java/io/github/dfa1/zstd/it/LargeMemoryMappedFileTest.java diff --git a/benchmark/README.md b/benchmark/README.md index e8e878d..f7d56fa 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -29,6 +29,14 @@ minimum): Payloads (`BenchData`) are deterministic, ~3x-compressible text so the ratios are realistic rather than all-zeros or random noise. +Plus a large-file suite, kept separate because it writes real payload files to +disk and is disk-/time-heavy (see below): + +- `LargeFileBenchmark` — mmap + `ZstdCompressStream` vs. `zstd-jni`'s classic + `ZstdOutputStream` path, at sizes from 4 MiB to 10 GiB. This is the JMH-backed + source for the mmap-vs-`zstd-jni` numbers in + [docs/zero-copy.md](../docs/zero-copy.md). + ## Build ```bash @@ -41,8 +49,8 @@ Produces a self-contained `benchmark/target/benchmarks.jar`. The host's native ## Run ```bash -# everything (full warmup/measurement — takes a few minutes) -java -jar benchmark/target/benchmarks.jar +# everything except LargeFileBenchmark (full warmup/measurement — a few minutes) +java -jar benchmark/target/benchmarks.jar -e LargeFileBenchmark # one suite, one size java -jar benchmark/target/benchmarks.jar CompressBenchmark -p size=1048576 @@ -54,6 +62,19 @@ java -jar benchmark/target/benchmarks.jar -f 1 -wi 1 -i 3 -p size=65536 `--enable-native-access=ALL-UNNAMED` is applied to forked JVMs via `@Fork`, so no extra flags are needed. +**Always exclude or explicitly filter to `LargeFileBenchmark`** — unlike the +suites above, a full run writes payload files up to 10 GiB (cached under +`${java.io.tmpdir}/zstd-java-bench-large-files/`, reused across variants and +reruns but never deleted automatically) and takes tens of minutes: + +```bash +# full large-file suite (all 5 sizes, 4 MiB–10 GiB — tens of minutes, ~16 GiB disk) +java -jar benchmark/target/benchmarks.jar LargeFileBenchmark + +# one size only +java -jar benchmark/target/benchmarks.jar LargeFileBenchmark -p sizeLabel=64MiB +``` + ## Reading results Throughput is `ops/ms` (higher is better). Compare rows at the same `(size)`: diff --git a/benchmark/src/main/java/io/github/dfa1/zstd/bench/LargeFileBenchmark.java b/benchmark/src/main/java/io/github/dfa1/zstd/bench/LargeFileBenchmark.java new file mode 100644 index 0000000..ca507a9 --- /dev/null +++ b/benchmark/src/main/java/io/github/dfa1/zstd/bench/LargeFileBenchmark.java @@ -0,0 +1,292 @@ +package io.github.dfa1.zstd.bench; + +import static java.lang.foreign.ValueLayout.ADDRESS; +import static java.lang.foreign.ValueLayout.JAVA_INT; +import static java.lang.foreign.ValueLayout.JAVA_LONG; + +import io.github.dfa1.zstd.ZstdCompressStream; +import io.github.dfa1.zstd.ZstdEndDirective; +import io.github.dfa1.zstd.ZstdOutputStream; +import io.github.dfa1.zstd.ZstdStreamResult; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.lang.foreign.Arena; +import java.lang.foreign.FunctionDescriptor; +import java.lang.foreign.Linker; +import java.lang.foreign.MemorySegment; +import java.lang.invoke.MethodHandle; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.channels.WritableByteChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.TimeUnit; +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Level; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.Warmup; + +/// JMH-backed successor to the ad hoc `System.nanoTime` timing loops that +/// produced the mmap-vs-`zstd-jni` numbers in `docs/zero-copy.md`: whether +/// mapping a large file into one `MemorySegment` and streaming it through +/// [ZstdCompressStream] is actually faster than `zstd-jni`'s classic +/// `ZstdOutputStream`-over-`InputStream` path, and whether +/// `posix_madvise(WILLNEED)` changes the answer. +/// +/// Four variants at each size, all reading/writing in `CHUNK`-sized (8 MiB) +/// steps so buffer size is not a confound between them: +/// +/// - `zstdJavaMmap` — this library, `FileChannel.map` into a `MemorySegment`, +/// streamed through [ZstdCompressStream]. No I/O hint. +/// - `zstdJavaMmapWillNeed` — same, plus `posix_madvise(WILLNEED)` on the +/// mapped segment right after mapping. +/// - `zstdJavaStream` — this library's own [ZstdOutputStream], fed from a +/// `byte[]` read loop. Not mmap — isolates whether this library's own +/// streaming API is competitive independent of the mmap question. +/// - `zstdJniStream` — `zstd-jni`'s `ZstdOutputStream`, fed from the same +/// read loop. +/// +/// There is no `FileInputStream#transferTo` variant: `transferTo`'s buffer +/// size is not configurable (a hardcoded 8 KiB when the destination isn't a +/// `FileOutputStream`, as `ZstdOutputStream` never is), so it cannot be made +/// to match `CHUNK` — comparing it here would confound "different code path" +/// with "different buffer size." +/// +/// `posix_madvise` is POSIX-only (Linux, macOS); on Windows +/// `zstdJavaMmapWillNeed` silently degrades to plain `zstdJavaMmap` — expected, +/// not a bug, since there is no direct equivalent (`PrefetchVirtualMemory` is a +/// structurally different API). +/// +/// Payload files are large (up to 10 GiB) and expensive to generate, so they +/// are cached in `${java.io.tmpdir}/zstd-java-bench-large-files/.bin` and +/// reused across variants and reruns instead of being rewritten per trial. +/// Nothing deletes that directory automatically — clean it up by hand once +/// done. Because of the size range, a full run is disk- and time-heavy (tens +/// of minutes, several GiB free disk); always invoke with an explicit filter, +/// never as part of an unfiltered `java -jar benchmarks.jar`: +/// +/// ``` +/// java -jar benchmark/target/benchmarks.jar LargeFileBenchmark -p sizeLabel=64MiB +/// ``` +@SuppressWarnings("restricted") // downcallHandle is a restricted FFM method +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.MILLISECONDS) +@State(Scope.Thread) +@Fork(value = 1, jvmArgsAppend = "--enable-native-access=ALL-UNNAMED") +@Warmup(iterations = 2) +@Measurement(iterations = 3) +public class LargeFileBenchmark { + + private static final int CHUNK = 8 * 1024 * 1024; + private static final int POSIX_MADV_WILLNEED = 3; + + private static final Path CACHE_DIR = + Path.of(System.getProperty("java.io.tmpdir"), "zstd-java-bench-large-files"); + + private static final Map SIZES = Map.of( + "4MiB", 4L * 1024 * 1024, + "64MiB", 64L * 1024 * 1024, + "2.25GiB", (long) Integer.MAX_VALUE + 256L * 1024 * 1024, + "4GiB", 4L * 1024 * 1024 * 1024, + "10GiB", 10L * 1024 * 1024 * 1024); + + private static final Optional POSIX_MADVISE = lookupPosixMadvise(); + + @Param({"4MiB", "64MiB", "2.25GiB", "4GiB", "10GiB"}) + private String sizeLabel; + + @Param({"3"}) + private int level; + + private Path file; + private long size; + + @Setup(Level.Trial) + public void setup() throws IOException { + size = SIZES.get(sizeLabel); + file = CACHE_DIR.resolve(sizeLabel + ".bin"); + writeIfMissing(file, size); + } + + @Benchmark + public long zstdJavaMmap() throws IOException { + return compressMmap(false); + } + + @Benchmark + public long zstdJavaMmapWillNeed() throws IOException { + return compressMmap(true); + } + + @Benchmark + public long zstdJavaStream() throws IOException { + CountingOutputStream sink = new CountingOutputStream(); + try (InputStream in = Files.newInputStream(file); + ZstdOutputStream out = new ZstdOutputStream(sink, level)) { + copy(in, out); + } + return sink.count(); + } + + @Benchmark + public long zstdJniStream() throws IOException { + CountingOutputStream sink = new CountingOutputStream(); + try (InputStream in = Files.newInputStream(file); + com.github.luben.zstd.ZstdOutputStream out = + new com.github.luben.zstd.ZstdOutputStream(sink, level)) { + copy(in, out); + } + return sink.count(); + } + + private long compressMmap(boolean advise) throws IOException { + try (FileChannel channel = FileChannel.open(file, StandardOpenOption.READ); + Arena arena = Arena.ofConfined(); + ZstdCompressStream cs = new ZstdCompressStream(level); + CountingChannel sink = new CountingChannel()) { + MemorySegment mapped = channel.map(FileChannel.MapMode.READ_ONLY, 0, size, arena); + if (advise) { + madviseWillNeed(mapped); + } + MemorySegment dst = arena.allocate(CHUNK); + ByteBuffer dstBuf = dst.asByteBuffer(); + long srcOff = 0; + ZstdStreamResult r; + do { + r = cs.compress(dst, mapped.asSlice(srcOff), ZstdEndDirective.END); + srcOff += r.bytesConsumed(); + dstBuf.clear().limit((int) r.bytesProduced()); + sink.write(dstBuf); + } while (!r.isComplete()); + return sink.count(); + } + } + + private static void copy(InputStream in, OutputStream out) throws IOException { + byte[] buffer = new byte[CHUNK]; + int n; + while ((n = in.read(buffer)) != -1) { + out.write(buffer, 0, n); + } + } + + private static void madviseWillNeed(MemorySegment mapped) { + if (POSIX_MADVISE.isEmpty()) { + return; + } + try { + int rc = (int) POSIX_MADVISE.orElseThrow() + .invokeExact(mapped, mapped.byteSize(), POSIX_MADV_WILLNEED); + if (rc != 0) { + System.err.println("posix_madvise(WILLNEED) failed rc=" + rc); + } + } catch (Throwable t) { + throw new RuntimeException("posix_madvise failed", t); + } + } + + private static Optional lookupPosixMadvise() { + return Linker.nativeLinker().defaultLookup().find("posix_madvise") + .map(addr -> Linker.nativeLinker().downcallHandle( + addr, FunctionDescriptor.of(JAVA_INT, ADDRESS, JAVA_LONG, JAVA_INT))); + } + + private static void writeIfMissing(Path file, long size) throws IOException { + if (Files.exists(file) && Files.size(file) == size) { + return; + } + Files.createDirectories(file.getParent()); + System.out.println("[LargeFileBenchmark] writing " + size + " bytes to " + file); + byte[] chunk = new byte[CHUNK]; + try (FileChannel channel = FileChannel.open(file, + StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING)) { + long written = 0; + while (written < size) { + int len = (int) Math.min(CHUNK, size - written); + for (int i = 0; i < len; i++) { + chunk[i] = expectedByteAt(written + i); + } + ByteBuffer buffer = ByteBuffer.wrap(chunk, 0, len); + while (buffer.hasRemaining()) { + channel.write(buffer); + } + written += len; + } + } + System.out.println("[LargeFileBenchmark] done writing " + file); + } + + private static byte expectedByteAt(long offset) { + return (byte) ('a' + (offset % 8)); + } + + /// Counts bytes without retaining them — the compressed-output sink for the + /// `byte[]`-fed variants. The returned count is also each `@Benchmark` + /// method's return value, which is what stops the JIT from proving the + /// whole compression call is dead code and eliminating it. + private static final class CountingOutputStream extends OutputStream { + + private long count; + + @Override + public void write(int b) { + count++; + } + + @Override + public void write(byte[] b, int off, int len) { + count += len; + } + + long count() { + return count; + } + } + + /// Counts bytes without retaining them — the compressed-output sink for + /// [#compressMmap]. A [WritableByteChannel] rather than an [OutputStream] + /// so the native `dst` segment can be handed over as a direct + /// [ByteBuffer] view with no heap `byte[]` bounce on the output side, + /// matching the zero-copy path this benchmark exists to measure. + private static final class CountingChannel implements WritableByteChannel { + + private long count; + private boolean open = true; + + // Always drains src fully in one call — unlike a real channel, so + // callers don't need the usual while (hasRemaining()) write() loop. + @Override + public int write(ByteBuffer src) { + int n = src.remaining(); + src.position(src.limit()); + count += n; + return n; + } + + @Override + public boolean isOpen() { + return open; + } + + @Override + public void close() { + open = false; + } + + long count() { + return count; + } + } +} diff --git a/docs/zero-copy.md b/docs/zero-copy.md index c641433..5fee542 100644 --- a/docs/zero-copy.md +++ b/docs/zero-copy.md @@ -51,7 +51,8 @@ to **mmap-slice → arena** (no boundary copy). ## Secondary wins -- **Zero GC** — off-heap, no allocation churn in a scan hot loop. +- **Zero GC** — off-heap, no allocation churn in a scan hot loop (measured, + not just asserted — see [the full reference table](#full-reference-every-variant-every-size-one-run)). - **No 2 GiB cap** — `byte[]` maxes at `Integer.MAX_VALUE`; segments are `long`-indexed. - **Lifetime safety** — bounds-checked, tied to a confined `Arena`; the same @@ -59,6 +60,250 @@ to **mmap-slice → arena** (no boundary copy). - **Typed reads** — read `JAVA_LONG` / `JAVA_DOUBLE` straight off the decompressed segment with no re-wrap. +## The 2 GiB cap: capability vs. throughput + +The "no 2 GiB cap" line above is a **capability** claim, and it is unconditional: +`FileChannel.map(mode, pos, size, Arena)` is `long`-indexed and maps a file of +any size in one call, on every OS this library supports. The reference +`zstd-jni` binding cannot do this at all — its entire zero-copy surface is +`java.nio.ByteBuffer`-based, and the classic 3-arg `FileChannel.map(mode, pos, +size)` it depends on throws `IllegalArgumentException` past +`Integer.MAX_VALUE`. There is no workaround inside zstd-jni; the caller would +have to hand-roll mapping the file in sub-2 GiB windows. This is proven by a +runnable test — see [below](#proof-largememorymappedfiletest). + +Whether that mapping is also **faster** than zstd-jni's fallback (a classic +`ZstdOutputStream` over `FileInputStream`, copying through a heap `byte[]`) is +a separate question, and the honest answer is: it depends on OS and size, not +a blanket yes. The macOS column below is JMH-backed +([`LargeFileBenchmark`](../benchmark/src/main/java/io/github/dfa1/zstd/bench/LargeFileBenchmark.java), +1 fork, 2 warmup + 3 measurement iterations — a quick, directional cut per +[ADR 0012](../adr/0012-benchmark-methodology.md), not the higher-fork/iteration +publication-grade run, but real iteration statistics with computed error, not +a hand-rolled timing loop). Linux and Windows have not been re-measured with +JMH yet and still carry the original ad hoc numbers — treat those two columns +as more provisional than the macOS one. + +Single-threaded at zstd's default level (3), comparing this library's +mmap-and-stream path (no I/O hint) against zstd-jni's stream path, same +synthetic, highly-compressible file: + +| size | macOS (Apple Silicon) | Linux (CI, x86_64) | Windows (CI, x86_64) | +|---|---|---|---| +| 4 MiB | roughly a wash, mmap slightly ahead | not measured | not measured | +| 64 MiB | mmap ~28% faster | not measured | not measured | +| 2.25 GiB | mmap ~21% faster | mmap ~2x faster | jni ~30% faster | +| 4 GiB | mmap ~23% faster | mmap ~2x faster | jni ~25% faster | +| 10 GiB | jni ~41% faster | not measured | not measured | + +Two things stand out: + +- **On macOS, the win reverses with scale** *if you map naively* — mmap is + clearly ahead from 64 MiB through 4 GiB, then loses by 10 GiB. This turns + out to be softenable, not fundamental — see + [below](#the-macos-reversal-was-a-missing-madvise-hint) — though the fix + closes most of the gap rather than flipping it outright. +- **On Windows, mmap never won** in the sizes tested, not even at 2.25 GiB. + zstd-jni's classic `ReadFile`-backed stream path was consistently faster. + Windows' memory-mapped-file implementation (`MapViewOfFile` under the hood) + is apparently costlier relative to buffered reads than on Linux or macOS. +- **On Linux, mmap won cleanly at both sizes tested**, by roughly 2x, with no + sign of the macOS-style reversal in this range. + +Caveats, stated plainly: the Linux and Windows cells are still single-CI-run, +ad hoc timing-loop measurements — directional, not authoritative, and several +were never measured (marked "not measured" rather than assumed). The macOS +column is JMH-backed but still a single machine and a low-iteration cut, not +the fork/iteration counts this project uses for a publication-grade claim +(see `benchmark/`). If throughput matters for your use case, benchmark your +own workload, OS, and file size. + +### The macOS reversal was a missing `madvise` hint + +`FileChannel.map()` — both the `ByteBuffer` and `MemorySegment` overloads — +just calls `mmap()`/`MapViewOfFile` under the hood and nothing else. It never +tells the OS how the mapping will be accessed, so the kernel falls back to its +default readahead heuristic. For a large, purely-sequential scan (exactly what +a one-shot compress of an mmap'd file is), that default turned out to be a +real cost on macOS: isolating pure I/O with zstd removed from the loop +entirely (mmap-and-touch-every-byte vs. `read()`-and-touch-every-byte, no +compression at all) showed mmap already losing to buffered `read()` well +before 10 GiB — by ~30% at 2.25 GiB and ~3x at 4 GiB. The compression numbers +above only looked good at those sizes because the actual `ZSTD_compressStream2` +CPU work was masking a growing I/O penalty underneath it; by 10 GiB the masked +penalty had grown past what the compute side could hide. + +Calling `posix_madvise(addr, len, POSIX_MADV_WILLNEED)` on the mapped segment +right after mapping it — a raw FFM downcall to libc, the same mechanism this +library already uses for `libzstd` itself — narrows the gap substantially. +JMH-backed (`LargeFileBenchmark`, same run as above), single-threaded, default +level, same synthetic file, macOS, mmap path with vs. without the hint, +compared again to zstd-jni's stream path: + +| size | mmap (no hint) | mmap + `WILLNEED` | zstd-jni (stream) | +|---|---|---|---| +| 64 MiB | 7.9 ms | **6.5 ms** | 10.1 ms | +| 2.25 GiB | 305 ms | **242 ms** | 369 ms | +| 4 GiB | 540 ms | **429 ms** | 664 ms | +| 10 GiB | 2300 ms | 1671 ms | **1628 ms** | + +The hint consistently cuts mmap's own time by ~18-27% at every size measured. +From 64 MiB through 4 GiB that's on top of an already-large lead — advised +mmap beats zstd-jni's stream path by roughly 50-57% there. At 10 GiB it closes +nearly all of the gap but not quite: 1671 ms vs. 1628 ms is a **statistical +tie** (well within each run's confidence interval), not the ~24% mmap win an +earlier, ad hoc (non-JMH) measurement of this same comparison had suggested. + +That earlier number came from a `System.nanoTime` loop with no warmup and no +iteration statistics — and, it turned out once this was rebuilt as a real JMH +benchmark, an accidental extra `byte[]` copy on the compressed-*output* side +of the mmap path (copying the native destination buffer into a heap array +before handing it to the sink, instead of writing the native buffer directly). +That copy pessimized mmap specifically, on top of everything else the ad hoc +harness got wrong. Fixing it and re-measuring properly moved the mmap numbers +considerably (faster at every size below 10 GiB than the ad hoc run ever +showed) without changing the qualitative shape: **mmap, especially advised, +wins clearly up to 4 GiB; at 10 GiB it's a tie with zstd-jni's stream path, +not a win.** For reference, this library's own non-mmap streaming path +(`ZstdOutputStream`, no `MemorySegment` involved) tracks zstd-jni's stream +path closely at every size (1737 ms vs. 1628 ms at 10 GiB) — confirming the +10 GiB story is specifically about mmap at that scale, not about the FFM +binding generally. + +Two things keep even these corrected numbers from being a settled conclusion: +it is still single-machine measurement (rerun on your own host before +quoting), and the hint has **only been tried on macOS**. Linux exposes the same +POSIX `posix_madvise`/`madvise` API but is untested here — it's plausible +either way, since Linux already won without any hint and may have little +headroom left to gain. Windows has no direct equivalent — the closest +primitive, `PrefetchVirtualMemory`, is a structurally different API and +hasn't been tried at all. This library does +not currently call `madvise` anywhere; doing so — and deciding whether it +belongs as an internal default, an opt-in flag, or just a documented recipe +for callers — is open follow-up work, not a shipped feature. + +### Full reference: every variant, every size, one run + +The percentages elsewhere in this doc are computed from these raw numbers — +`LargeFileBenchmark`, `-prof gc`, single run: + +**Time (avgt, lower is better):** + +| size | `zstdJavaMmap` | `zstdJavaMmapWillNeed` | `zstdJavaStream` | `zstdJniStream` | +|---|---:|---:|---:|---:| +| 4 MiB | 0.49 ms | 0.46 ms | 0.65 ms | 0.63 ms | +| 64 MiB | 7.8 ms | 6.7 ms | 10.7 ms | 10.2 ms | +| 2.25 GiB | 298 ms | 247 ms | 359 ms | 364 ms | +| 4 GiB | 536 ms | 433 ms | 668 ms | 645 ms | +| 10 GiB | 2349 ms | 1692 ms | 1751 ms | 1613 ms | + +**Allocation (`gc.alloc.rate.norm`, bytes/op):** + +| size | `zstdJavaMmap` | `zstdJavaMmapWillNeed` | `zstdJavaStream` | `zstdJniStream` | +|---|---:|---:|---:|---:| +| 4 MiB | 1,117 B | 1,156 B | 8,521,270 B | 8,520,896 B | +| 64 MiB | 1,424 B | 1,337 B | 8,521,312 B | 8,520,919 B | +| 2.25 GiB | 4,867 B | 4,286 B | 8,525,055 B | 8,521,162 B | +| 4 GiB | 2,977 B | 2,660 B | 8,522,406 B | 8,521,350 B | +| 10 GiB | 2,855 B | 2,624 B | 8,522,471 B | 8,521,912 B | + +mmap is allocation-free at every size (a few KB of bookkeeping, never the +payload); the streaming variants allocate a constant ~8.1 MiB/op regardless +of file size — because `copy()`'s read buffer is a fixed 8 MiB `CHUNK`, not +sized to the file. That constant shows up starkly in `gc.count`: at 4 MiB, +where each op takes ~0.6 ms, JMH packs thousands of ops into a single +measurement window, and `zstdJavaStream` racked up 908 GC collections +(~349 ms total) in that window — compressing a 4 MiB file that repeatedly +allocates-and-discards an 8 MiB array. By 10 GiB, where each op is +multi-second, that same fixed allocation costs a single GC (~1-2 ms) — +negligible. This is a real, measured number, not swept under the rug — but +it's an artifact of this benchmark holding buffer size constant across sizes +for a fair cross-variant comparison, not something a size-aware production +implementation would do. + +**Machine:** Apple M5, 32 GB RAM, macOS 26.5.2. JDK 25.0.2 (Zulu +`25.0.2+10-LTS`). zstd-jni 1.5.7-11 (bundles zstd 1.5.7, matching this +library's build). Level 3 (zstd default). 1 fork, 2 warmup + 3 measurement +iterations — a quick, directional cut per +[ADR 0012](../adr/0012-benchmark-methodology.md), not a publication-grade run. + +**Reproduce:** + +```bash +./mvnw -q -pl benchmark -am package -DskipTests +java -jar benchmark/target/benchmarks.jar LargeFileBenchmark -prof gc +``` + +Writes payload files up to 10 GiB under +`${java.io.tmpdir}/zstd-java-bench-large-files/` (cached across variants and +reruns, never deleted automatically — see +[benchmark/README.md](../benchmark/README.md)) and takes 15-30 minutes +depending on machine load. Always filter or exclude `LargeFileBenchmark` +explicitly — never run it as part of an unfiltered `benchmarks.jar` +invocation. + +### Proof: `LargeMemoryMappedFileTest` + +`integration-tests/.../LargeMemoryMappedFileTest` is a runnable, disabled-by- +default test that proves the *capability* claim: it writes a file just over +the `Integer.MAX_VALUE`-byte boundary, shows the classic `ByteBuffer` mapping +call rejects it, shows the FFM `MemorySegment` mapping call succeeds and +addresses bytes past the boundary correctly, and round-trips the whole file +through this library's zero-copy streaming compressor — plus a companion test +showing zstd-jni's stream API can still compress a file this large (just not +zero-copy map it) and that the frame it produces decodes correctly through +this library. It is gated behind a system property, since it writes several +gigabytes of real files to disk and is too slow/disk-hungry for a plain `mvn +test`/`mvn verify` or CI: + +``` +./mvnw -pl integration-tests test -Dzstd.test.large=true -Dtest=LargeMemoryMappedFileTest +``` + +## Is the extra code worth it? + +The capability and throughput numbers above come from two codepaths that +don't tell the same story: + +- **This library's own `ZstdOutputStream`** (`zstdJavaStream` in the + benchmark) is exactly as easy to use as zstd-jni's — a five-line + try-with-resources over a plain `InputStream`. But it is **not zero-copy**: + its `write(byte[], ...)` still does a `MemorySegment.copy` from the heap + into a native staging buffer per chunk, the same tax described in the + [honest caveat](#honest-caveat) below. It isn't faster than zstd-jni's + equivalent either — 1713 ms vs. 1616 ms at 10 GiB, a wash. +- **The raw `MemorySegment`/`ZstdCompressStream` path** (what + `zstdJavaMmap`/`zstdJavaMmapWillNeed` actually do) is the one that's + genuinely zero-copy — no heap bounce anywhere. It's also meaningfully more + code: a manual `Arena`, a hand-driven `mmap → compress → drain` loop, and + your own buffer/channel plumbing to get compressed bytes anywhere other + than an in-memory sink, since `ZstdCompressStream` works in + `MemorySegment`s, not `java.io` streams. `compressMmap` in + [`LargeFileBenchmark`](../benchmark/src/main/java/io/github/dfa1/zstd/bench/LargeFileBenchmark.java) + is ~15 lines including a custom `WritableByteChannel` sink, against the + five-line `ZstdOutputStream` snippet above. + +That harder path is also the one whose throughput edge shrinks to a +statistical tie right around the size (10 GiB) where the "no 2 GiB cap" +capability claim starts to matter most — below 2 GiB, zstd-jni's own +`ByteBuffer`-based zero-copy surface works fine, so the capability gap only +opens up exactly where, per the numbers above, the speed edge is closing. +**Mapping a file from scratch purely to hand it to zstd is not obviously +worth the extra code** on this data: you pay real `Arena`/`MemorySegment` +complexity for a result that, at large scale, roughly matches what five +lines of `ZstdOutputStream` already gets you. + +Where the complexity *is* worth paying: when the `MemorySegment` is +**already in your hand** for reasons that have nothing to do with zstd — the +memory-mapped-reader case from +[above](#when-it-actually-pays-off-not-always) (Vortex-style: the file is +mapped because that's how the reader works, not because compression asked +for it). That caller is managing `Arena`/`MemorySegment` lifetimes in their +own code regardless; handing zstd a slice of a segment they already have is +a small marginal addition, not a 15-line tax paid from zero. The +benchmark's own scenario — mmap-from-scratch, compress, discard — is the +case this project has the least evidence for recommending. + ## Honest caveat If the caller hands you a heap `byte[]` (the aircompressor fallback path, or diff --git a/integration-tests/src/test/java/io/github/dfa1/zstd/it/LargeMemoryMappedFileTest.java b/integration-tests/src/test/java/io/github/dfa1/zstd/it/LargeMemoryMappedFileTest.java new file mode 100644 index 0000000..26be312 --- /dev/null +++ b/integration-tests/src/test/java/io/github/dfa1/zstd/it/LargeMemoryMappedFileTest.java @@ -0,0 +1,263 @@ +package io.github.dfa1.zstd.it; + +import com.github.luben.zstd.ZstdOutputStream; +import io.github.dfa1.zstd.ZstdCompressStream; +import io.github.dfa1.zstd.ZstdDecompressStream; +import io.github.dfa1.zstd.ZstdEndDirective; +import io.github.dfa1.zstd.ZstdStreamResult; +import org.assertj.core.api.ThrowableAssert.ThrowingCallable; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.junit.jupiter.api.io.TempDir; + +import java.io.BufferedInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.lang.foreign.Arena; +import java.lang.foreign.MemorySegment; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; + +import static java.lang.foreign.ValueLayout.JAVA_BYTE; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/// Proves the zero-copy claim from `docs/zero-copy.md` that the `MemorySegment` +/// API removes the 2 GiB cap that `byte[]` / `ByteBuffer` impose, and that this +/// project's segment-based streaming compressor round-trips a file larger than +/// [Integer#MAX_VALUE] bytes read straight off a single memory mapping — and +/// contrasts that with the reference `zstd-jni` binding, whose zero-copy surface +/// is entirely `ByteBuffer`-based and hits the same `int` cap with no way around +/// it (it falls back to the classic heap-copying stream API instead). +/// +/// This test creates a file just over 2 GiB on disk, memory-maps it, and streams +/// the whole thing through compress/decompress, so it is slow and disk-hungry — +/// far too heavy for a plain `mvn test`/`mvn verify` or CI. It is therefore gated +/// behind the `zstd.test.large` system property and skipped unless it is `true`. +/// +/// Run it explicitly with: +/// +/// ``` +/// ./mvnw -pl integration-tests test -Dzstd.test.large=true -Dtest=LargeMemoryMappedFileTest +/// ``` +@EnabledIfSystemProperty(named = "zstd.test.large", matches = "true") +class LargeMemoryMappedFileTest { + + /// A size strictly greater than [Integer#MAX_VALUE], so the `int`-indexed + /// [ByteBuffer] path cannot address it. About 2.25 GiB. + private static final long FILE_SIZE = (long) Integer.MAX_VALUE + 256L * 1024 * 1024; + + /// Chunk used both to write the file and to feed the streaming buffers. + private static final int CHUNK = 8 * 1024 * 1024; + + @TempDir + private static Path tempDir; + + private static Path file; + + /// Written once and shared by every test in this class — regenerating a + /// 2.25 GiB file per test would double the already-heavy I/O for no benefit, + /// since every test here only reads it. + @BeforeAll + static void writeSharedFile() throws IOException { + file = tempDir.resolve("large-2gib.bin"); + writeDeterministicFile(file); + } + + /// Deterministic content: every byte is a pure function of its absolute file + /// offset, so any offset (including past 2 GiB) can be regenerated for + /// verification without keeping a multi-gigabyte reference buffer around. + private static byte expectedByteAt(long offset) { + return (byte) ('a' + (offset % 8)); + } + + @Test + void segmentMappingBeatsByteBufferAndRoundTripsPast2Gib() throws IOException { + // Given the shared file, just over the Integer.MAX_VALUE byte boundary + try (FileChannel channel = FileChannel.open(file, StandardOpenOption.READ); + Arena arena = Arena.ofConfined()) { + + // When the classic int-indexed ByteBuffer overload is asked to map + // more than Integer.MAX_VALUE bytes + ThrowingCallable byteBufferMap = + () -> channel.map(FileChannel.MapMode.READ_ONLY, 0, FILE_SIZE); + + // Then it rejects the request outright + assertThatThrownBy(byteBufferMap).isInstanceOf(IllegalArgumentException.class); + + // When the long-indexed FFM overload maps the same range in one call + MemorySegment mapped = channel.map(FileChannel.MapMode.READ_ONLY, 0, FILE_SIZE, arena); + + // Then the whole file is addressable from a single segment + assertThat(mapped.byteSize()).isEqualTo(FILE_SIZE); + + // And reads at offsets past Integer.MAX_VALUE resolve correctly, + // proving addressing beyond the int boundary + long[] probes = { + (long) Integer.MAX_VALUE, + (long) Integer.MAX_VALUE + 1, + (long) Integer.MAX_VALUE + 123_456_789L, + FILE_SIZE - 1 + }; + for (long offset : probes) { + assertThat(mapped.get(JAVA_BYTE, offset)).isEqualTo(expectedByteAt(offset)); + } + + // When the mapped segment is streamed through the zero-copy compressor + byte[] frame = compressFromSegment(arena, mapped); + + // Then decompressing verifies every byte in place against the same + // offset -> byte function, without materializing 2.25 GiB anywhere + long produced = verifyDecompress(arena, frame); + assertThat(produced).isEqualTo(FILE_SIZE); + } + } + + @Test + void zstdJniCannotZeroCopyMapTheWholeFileButItsStreamApiStillCompressesIt() throws IOException { + // Given the shared file, and zstd-jni's zero-copy surface — which is + // entirely java.nio.ByteBuffer, backed by the same 3-arg FileChannel.map + try (FileChannel channel = FileChannel.open(file, StandardOpenOption.READ)) { + // When it is asked to map the whole file for a zero-copy compress call + ThrowingCallable byteBufferMap = + () -> channel.map(FileChannel.MapMode.READ_ONLY, 0, FILE_SIZE); + + // Then it hits the exact same int cap zstd-jni's ByteBuffer API cannot + // work around: there is no long-indexed mapping in zstd-jni to fall + // back to, unlike the FFM MemorySegment path above + assertThatThrownBy(byteBufferMap).isInstanceOf(IllegalArgumentException.class); + } + + // When zstd-jni instead falls back to its classic stream API, copying + // through a heap byte[] buffer rather than mapping the file at all + byte[] frame = compressWithZstdJni(file); + + // Then it does successfully compress the whole file, and the frame it + // produced decodes correctly through this library's own zero-copy + // streaming decompressor — proving real format interop at this size + try (Arena arena = Arena.ofConfined()) { + long produced = verifyDecompress(arena, frame); + assertThat(produced).isEqualTo(FILE_SIZE); + } + } + + /// Compresses `file` with the reference `zstd-jni` binding via its classic + /// [ZstdOutputStream], reading through a reused heap `byte[]` buffer — the + /// heap-copying path zstd-jni falls back to since it has no `long`-indexed + /// zero-copy alternative. + private static byte[] compressWithZstdJni(Path file) throws IOException { + ByteArrayOutputStream frame = new ByteArrayOutputStream(); + try (InputStream in = new BufferedInputStream(Files.newInputStream(file), CHUNK); + ZstdOutputStream out = new ZstdOutputStream(frame, 1)) { + byte[] buffer = new byte[CHUNK]; + int n; + while ((n = in.read(buffer)) != -1) { + out.write(buffer, 0, n); + } + } + return frame.toByteArray(); + } + + /// Writes [#FILE_SIZE] bytes to `file` in [#CHUNK]-sized batches. Each byte is + /// [#expectedByteAt(long)] of its absolute offset, so no random source and no + /// full-file buffer are needed. + private static void writeDeterministicFile(Path file) throws IOException { + byte[] chunk = new byte[CHUNK]; + try (FileChannel channel = FileChannel.open( + file, StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { + long written = 0; + while (written < FILE_SIZE) { + int len = (int) Math.min(CHUNK, FILE_SIZE - written); + for (int i = 0; i < len; i++) { + chunk[i] = expectedByteAt(written + i); + } + ByteBuffer buffer = ByteBuffer.wrap(chunk, 0, len); + while (buffer.hasRemaining()) { + channel.write(buffer); + } + written += len; + } + } + } + + /// Streams the whole `source` segment through [ZstdCompressStream], always + /// under [ZstdEndDirective#END], slicing from a running offset to the end of + /// the segment each call and looping until the frame is complete — the same + /// shape as `ZstdSegmentStreamTest.Chunked`. `ZstdStreamBuffer#set` only + /// points a native struct at `src`; it never copies, so handing the entire + /// remaining (multi-gigabyte) slice in each call is free — consumption per + /// call is bounded by `dst`'s capacity, not by the slice length. The payload + /// is highly compressible, so the resulting frame is small enough to + /// accumulate in memory. + private static byte[] compressFromSegment(Arena arena, MemorySegment source) { + ByteArrayOutputStream frame = new ByteArrayOutputStream(); + MemorySegment dst = arena.allocate(CHUNK); + byte[] scratch = new byte[CHUNK]; + try (ZstdCompressStream cs = new ZstdCompressStream(1)) { + long srcOff = 0; + long total = source.byteSize(); + ZstdStreamResult r; + do { + MemorySegment srcSlice = source.asSlice(srcOff, total - srcOff); + r = cs.compress(dst, srcSlice, ZstdEndDirective.END); + srcOff += r.bytesConsumed(); + drain(dst, r.bytesProduced(), scratch, frame); + } while (!r.isComplete()); + } + return frame.toByteArray(); + } + + /// Copies the first `produced` bytes of `dst` into `frame` via a reused + /// `scratch` array, avoiding a fresh allocation per streaming step. + private static void drain(MemorySegment dst, long produced, byte[] scratch, ByteArrayOutputStream frame) { + int n = (int) produced; + MemorySegment.copy(dst, JAVA_BYTE, 0, scratch, 0, n); + frame.write(scratch, 0, n); + } + + /// Decompresses `frame` through [ZstdDecompressStream], verifying each output + /// chunk in place against [#expectedByteAt(long)] using a running absolute + /// output offset, so the ~2.25 GiB decompressed stream is never held in full. + /// + /// @return the total number of decompressed bytes produced + private static long verifyDecompress(Arena arena, byte[] frame) { + MemorySegment src = arena.allocate(frame.length); + MemorySegment.copy(frame, 0, src, JAVA_BYTE, 0, frame.length); + MemorySegment dst = arena.allocate(CHUNK); + + long outOff = 0; + try (ZstdDecompressStream ds = new ZstdDecompressStream()) { + long srcOff = 0; + ZstdStreamResult r; + do { + MemorySegment srcSlice = src.asSlice(srcOff, frame.length - srcOff); + r = ds.decompress(dst, srcSlice); + srcOff += r.bytesConsumed(); + outOff += verifyChunk(dst, r.bytesProduced(), outOff); + } while (srcOff < frame.length || !r.isComplete()); + } + return outOff; + } + + /// Checks that `produced` bytes at the front of `dst` match the deterministic + /// content starting at absolute output offset `outOff`. + /// + /// @return `produced`, so callers can advance the running offset + private static long verifyChunk(MemorySegment dst, long produced, long outOff) { + for (long i = 0; i < produced; i++) { + byte actual = dst.get(JAVA_BYTE, i); + byte expected = expectedByteAt(outOff + i); + if (actual != expected) { + assertThat(actual) + .as("byte at output offset %d", outOff + i) + .isEqualTo(expected); + } + } + return produced; + } +}