Summary
InspectorTree.build()'s usedEncodings set (surfaced by the CLI's inspect command) only reports the root ArrayNode's encoding for each Flat layout segment — it does not recursively walk into encodings nested inside that same segment's buffer. This makes encodings like vortex.fsst invisible in the inspector even when they're genuinely selected and used by the writer.
Repro
Import any dataset with a high-cardinality Utf8 column through ParquetImporter/VortexWriter with WriteOptions.cascading(3) (e.g. NYC 311 complaints' "Incident Address" column). The cost-based dispatch in CascadingCompressor correctly picks vortex.fsst for such columns (confirmed by instrumenting MaskedEncodingEncoder.encodeValues directly — the real write path logs picked=vortex.fsst, and the on-disk size matches FSST's expected output almost exactly).
Yet java -jar vortex-cli-*.jar inspect <file> shows:
Used encodings: vortex.masked, vortex.struct
No vortex.fsst anywhere, and the Layout: tree's bracketed encoding list for that column also omits it — even though the values genuinely are FSST-compressed.
Root cause
inspector/src/main/java/io/github/dfa1/vortex/inspect/InspectorTree.java's peekFlatRoot(MemorySegment seg, List<EncodingId> arraySpecs) reads only the root FbsArrayNode of a Flat segment's trailing FlatBuffer:
FbsArrayNode root = fbArray.root();
...
return new Peek(arraySpecs.get(root.encoding()).id(), ArrayStats.fromFbs(root.stats()));
It never descends into root.children(). This is fine for encodings that are themselves separate layout nodes with their own segments (e.g. vortex.dict, whose dictionary-values child is a distinct Flat layout node, independently peekable) — but any encoding that lives purely inside one Flat segment's ArrayNode tree (FSST, ALP, bitpacked when not chunked separately, etc.) is invisible to this shallow peek.
Impact
- Purely a diagnostic/observability gap —
VortexReader.scan() decodes the full ArrayNode tree correctly regardless, so this does not affect correctness of reads.
- But it makes the CLI
inspect command (and anything built on InspectorTree.usedEncodings(), e.g. the JavaVsJniFsstBenchmark's verifyFsstSelected check) unreliable for auditing which encoding a real file's column actually uses whenever that encoding is nested rather than a top-level layout wrapper.
Expected
InspectorTree should recursively walk a Flat segment's full ArrayNode tree (not just its root) when computing usedEncodings/the per-node encoding list, so encodings like vortex.fsst are reported accurately. This likely needs peekFlatRoot (or a new sibling helper) to walk root.children() using the same arraySpecs resolution, accumulating every encoding id encountered rather than stopping at the root.
Where
inspector/src/main/java/io/github/dfa1/vortex/inspect/InspectorTree.java — peekFlatRoot, and whichever caller(s) build the per-Node usedEncodings set from that peek result.
Summary
InspectorTree.build()'susedEncodingsset (surfaced by the CLI'sinspectcommand) only reports the rootArrayNode's encoding for each Flat layout segment — it does not recursively walk into encodings nested inside that same segment's buffer. This makes encodings likevortex.fsstinvisible in the inspector even when they're genuinely selected and used by the writer.Repro
Import any dataset with a high-cardinality Utf8 column through
ParquetImporter/VortexWriterwithWriteOptions.cascading(3)(e.g. NYC 311 complaints' "Incident Address" column). The cost-based dispatch inCascadingCompressorcorrectly picksvortex.fsstfor such columns (confirmed by instrumentingMaskedEncodingEncoder.encodeValuesdirectly — the real write path logspicked=vortex.fsst, and the on-disk size matches FSST's expected output almost exactly).Yet
java -jar vortex-cli-*.jar inspect <file>shows:No
vortex.fsstanywhere, and theLayout:tree's bracketed encoding list for that column also omits it — even though the values genuinely are FSST-compressed.Root cause
inspector/src/main/java/io/github/dfa1/vortex/inspect/InspectorTree.java'speekFlatRoot(MemorySegment seg, List<EncodingId> arraySpecs)reads only the rootFbsArrayNodeof a Flat segment's trailing FlatBuffer:It never descends into
root.children(). This is fine for encodings that are themselves separate layout nodes with their own segments (e.g.vortex.dict, whose dictionary-values child is a distinct Flat layout node, independently peekable) — but any encoding that lives purely inside one Flat segment'sArrayNodetree (FSST, ALP, bitpacked when not chunked separately, etc.) is invisible to this shallow peek.Impact
VortexReader.scan()decodes the fullArrayNodetree correctly regardless, so this does not affect correctness of reads.inspectcommand (and anything built onInspectorTree.usedEncodings(), e.g. theJavaVsJniFsstBenchmark'sverifyFsstSelectedcheck) unreliable for auditing which encoding a real file's column actually uses whenever that encoding is nested rather than a top-level layout wrapper.Expected
InspectorTreeshould recursively walk a Flat segment's fullArrayNodetree (not just its root) when computingusedEncodings/the per-node encoding list, so encodings likevortex.fsstare reported accurately. This likely needspeekFlatRoot(or a new sibling helper) to walkroot.children()using the samearraySpecsresolution, accumulating every encoding id encountered rather than stopping at the root.Where
inspector/src/main/java/io/github/dfa1/vortex/inspect/InspectorTree.java—peekFlatRoot, and whichever caller(s) build the per-NodeusedEncodingsset from that peek result.