Skip to content

docs: zero-copy mmap-vs-jni story, JMH-backed#101

Merged
dfa1 merged 1 commit into
mainfrom
feature/large-file-benchmark
Jul 24, 2026
Merged

docs: zero-copy mmap-vs-jni story, JMH-backed#101
dfa1 merged 1 commit into
mainfrom
feature/large-file-benchmark

Conversation

@dfa1

@dfa1 dfa1 commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds benchmark/.../LargeFileBenchmark, a JMH benchmark comparing this library's mmap+MemorySegment path (with/without posix_madvise(WILLNEED)) against zstd-jni's classic streaming API, at sizes from 4 MiB to 10 GiB — replacing the ad hoc System.nanoTime scratch harnesses that produced the previously-committed numbers in docs/zero-copy.md.
  • Fixes a real bug the switch to JMH surfaced: the mmap path bounced compressed output through a heap byte[] before handing it to the sink, defeating the zero-copy point. Now uses a direct ByteBuffer view over the native dst segment + a WritableByteChannel sink. Verified byte-for-byte round-trip against the independent zstd CLI.
  • Adds integration-tests/.../LargeMemoryMappedFileTest, a disabled-by-default test proving the library's >2 GiB memory-mapping capability claim (gated behind -Dzstd.test.large=true, too heavy for CI).
  • Rewrites docs/zero-copy.md with the corrected, JMH-backed numbers (time + -prof gc allocation, full reference table with min–max ranges since JMH's 99.9% CI is unstable at n=3, machine specs, reproduce command), an honest account of what the ad hoc harness got wrong, and a new "Is the extra code worth it?" section.

Closes #95.

What the real numbers show (vs. the ad hoc ones they replace)

The corrected data confirms the shape of the earlier finding — mmap ahead at medium sizes, the madvise hint mattering more at 10 GiB — but at roughly a tenth the magnitude the ad hoc timing loops reported (4–5% swings at 10 GiB, not 24–45%). It also surfaces something the ad hoc numbers missed entirely: at 2.25–4 GiB, the WILLNEED hint is measurably slower than no hint at all, only becoming a net win again at 10 GiB. The one clean, decisive result across the board: at 4 MiB, mmap beats zstd-jni by ~2x — every measured iteration faster than every zstd-jni iteration — driven by fixed per-call overhead, not throughput.

Net conclusion for the new "Is the extra code worth it?" section: the zero-copy path is ~4x the code of the simple ZstdOutputStream API, and its throughput edge is decisive only at small files — at large scale it's a few percent either way. The real argument for the extra code at scale is the unconditional >2 GiB capability claim, not throughput.

Test plan

  • ./mvnw -pl zstd,integration-tests,benchmark -am clean verify — builds and tests clean
  • ./mvnw -pl benchmark validate / ./mvnw -pl zstd validate javadoc:javadoc — checkstyle/javadoc clean
  • Smoke-tested all 3 LargeFileBenchmark variants at small size (-p size=4194304 -wi 1 -i 1 -f 1)
  • Verified the zero-copy-output-fix round-trips byte-for-byte via the independent zstd CLI
  • Ran the full suite (all 5 sizes × 3 variants, with -prof gc) to completion; numbers written into the docs are from that real run, not fabricated

🤖 Generated with Claude Code

Adds benchmark/.../LargeFileBenchmark, a JMH benchmark comparing this
library's mmap + MemorySegment path (with/without posix_madvise(WILLNEED))
against zstd-jni's classic streaming API, at sizes from 4 MiB to 10 GiB -
replacing the ad hoc System.nanoTime scratch harnesses that produced the
previously-committed numbers in docs/zero-copy.md. Each @benchmark
invocation is one full-file compression, so this uses Mode.SingleShotTime
(JMH times individual invocations) rather than the Throughput mode the
other benchmarks use for tight loops. The source file for a given size is
generated once and cached under the system temp directory, shared across
all three variants' JMH forks rather than rewritten per variant.

Fixes a real bug the switch to JMH surfaced: the mmap path was bouncing
compressed output through a heap byte[] before handing it to the sink,
defeating the zero-copy point. Now uses a direct ByteBuffer view over the
native destination segment straight into a WritableByteChannel sink.
Verified byte-for-byte round-trip against the independent zstd CLI.

Also adds integration-tests/.../LargeMemoryMappedFileTest, a disabled-by-
default test proving the library's >2 GiB memory-mapping capability claim:
the classic ByteBuffer-based FileChannel.map overload rejects a file past
Integer.MAX_VALUE, the long-indexed MemorySegment overload maps it in one
call and addresses bytes past the boundary correctly, and the whole file
round-trips through this library's zero-copy streaming compressor - plus a
companion test showing zstd-jni's stream API can still compress a file this
large (just not zero-copy map it) and that the frame it produces decodes
correctly through this library.

Rewrites docs/zero-copy.md's throughput section with the corrected,
JMH-backed numbers (time + -prof gc allocation, full reference table with
min-max ranges since JMH's 99.9% CI is unstable at n=3, machine specs,
reproduce command). The corrected data confirms the shape of the earlier ad
hoc finding - mmap ahead at medium sizes, the madvise hint mattering more
at 10 GiB - but at roughly a tenth the magnitude the ad hoc timing loops
reported, plus a previously-unseen small reversal at 2.25-4 GiB where the
WILLNEED hint costs more than it saves. Adds an "Is the extra code worth
it?" section arguing the zero-copy MemorySegment path is real but ~4x more
code than the easy ZstdOutputStream API, and its throughput edge is only
decisive at small files - at large scale it's a few percent either way,
which is where the "no 2 GiB cap" capability claim (not throughput) becomes
the real argument for the extra code.

Independently reviewed: verified the mmap zero-copy fix round-trips
byte-for-byte via the zstd CLI, verified (empirically, by reproducing the
data generator and compressing both variants) that the benchmark's
repeated-8-MiB-chunk source data doesn't inflate the measured compression
ratio versus continuously-varying data, and fixed a real 3x-redundant-write
inefficiency in the benchmark's own setup by caching the generated source
file per size instead of regenerating it per variant.

Closes #95.
@dfa1
dfa1 force-pushed the feature/large-file-benchmark branch from fd74188 to a3a722a Compare July 24, 2026 07:26
@dfa1

dfa1 commented Jul 24, 2026

Copy link
Copy Markdown
Owner Author

Ran an independent review over the full diff and applied the findings that survived verification:

  • Fixed a real 3x-redundant-write inefficiency: LargeFileBenchmark's @Setup(Level.Trial) was regenerating an identical source file for every (benchmark method × size) combination, since each of the 3 variants runs in its own JMH fork — 30 GiB of redundant setup writes at the 10 GiB point alone. Now cached under the system temp directory keyed by size, generated once and reused across all three variants. (Smoke-testing the fix itself caught a real bug in the fix: the cache path needed CREATE added to writeCompressibleFile's FileChannel.openTRUNCATE_EXISTING alone requires the file to already exist, which the old code got away with because Files.createTempFile always pre-created it first. Verified the cache is actually reused via mtime staying unchanged across a full re-run.)
  • Tightened docs/zero-copy.md's "Is the extra code worth it?" recommendation — the n=3/single-machine qualifier was one sentence upstream of the actual recommendation; pulled it directly into that sentence.

Also verified two other candidates that came back clean (no change needed):

  • A claimed resource-leak in mmapCompress's try-with-resources — refuted; Java guarantees all four resources close regardless of where in the try block an exception originates.
  • Whether the benchmark's repeated-8-MiB-chunk source data (a shortcut vs. regenerating fresh content) inflates the measured compression ratio — refuted empirically, twice independently: reproduced the exact generator and compressed both the repeated-chunk and a continuously-varying control at the same size through both this library and the real zstd CLI; ratios matched within ~0.01%. At level 3, zstd's window (2 MiB) is smaller than the repeated chunk (8 MiB), so the compressor never gets to exploit the repeat.

Full build/tests/checkstyle/javadoc all green. Pushed as a3a722a.

@dfa1
dfa1 merged commit f6e2c5c into main Jul 24, 2026
1 check passed
@dfa1
dfa1 deleted the feature/large-file-benchmark branch July 24, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant