CompactProof: Deduplicate values by skunert · Pull Request #233 · paritytech/trie

skunert · 2026-07-07T15:12:17Z

Summary

encode_compact re-emits a shared detached value once per referencing node, causing proof size to grow with reference count instead of value count. Adds opt-in encode_compact_skip_duplicate_values to dedupe on encode. decode_compact_from_iter (and decode_compact) now always handle both old-style and deduplicated proofs correctly. Also fixes an off-by-one in decode_compact_from_iter's returned item count when the last node has an attached value.

Compatibility

encode_compact / decode_compact_from_iter behavior unchanged — new functions are additive.
Deduplicated proofs decode correctly with this crate's decoder, and with an unpatched old decoder into a hash-keyed MemoryDB (verified against 0.31.0) — but not into a PrefixedKey DB, which needs the value re-inserted per prefix.
Old decoder + hash-keyed DB: refcount for the shared value is undercounted (1 instead of N). Reads still work (get/contains only check rc > 0); This matters if you later do balanced insert/remove against that decoded DB.
decode_compact's item count is now correct when the last node has an attached value — previously it was off by one in that case.

encode_compact emits a detached value node once per referencing trie node, so a value shared by N keys is sent N times (paritytech/polkadot-sdk#12565). Add encode_compact_skip_duplicate_values to emit each distinct value once; repeats are emitted unmodified and stay decodable by existing hash-keyed decoders. For prefixed databases, decode_compact_from_iter_with_known_values re-inserts deduplicated values at every referencing position.

A trailing attached value node was not counted in the returned used-item count, so decoding concatenated encodings at that offset re-read the value bytes as a node.

trie_codec_proof only ran ExtensionLayout (inline values), so value detachment and deduplication were never fuzzed. Round-trip the deduplicating encoding into hash-keyed and prefixed databases, add a hashed-value target with heavily shared values, and a deterministic smoke test so the assertions run in CI without libFuzzer. Also fix the fuzz crate's stale memory-db path dependency version.

An old decoder inserts a deduplicated value once, so its refcount in a hash-keyed database understates the referencing nodes; a consumer consolidating removals could drop a still-referenced value. The re-inserting decoder restores exact parity. Assert full database equality (entries and refcounts, hash-keyed and prefixed) between the plain and deduplicated encodings.

bkchr

You can do the de-duplication on the node level and not just on the value level.

bkchr · 2026-07-09T10:45:25Z

+/// [`decode_compact_from_iter_with_known_values`]).
+pub fn encode_compact_skip_duplicate_values<L>(
+	db: &TrieDB<L>,
+	seen_value_hashes: &mut BTreeSet<Vec<u8>>,


Does someone requires this after the encoding?

We could use it for example here to deduplicate across child and main trie. Whether its really necessary to deduplicate across main and child is not super clear to me, but I can see use cases for this.

bkchr · 2026-07-09T10:45:51Z

+/// `seen_value_hashes` collects the emitted value hashes. Pass `&mut Default::default()` for a
+/// standalone encoding, or thread the same set across concatenated encodings (the matching decode
+/// calls must then thread a known-values map through
+/// [`decode_compact_from_iter_with_known_values`]).


Suggested change

/// `seen_value_hashes` collects the emitted value hashes. Pass `&mut Default::default()` for a

/// standalone encoding, or thread the same set across concatenated encodings (the matching decode

/// calls must then thread a known-values map through

/// [`decode_compact_from_iter_with_known_values`]).

/// `seen_value_hashes` collects the emitted value hashes.

Ty claude :P

What is the use case for "concatenated encodings"? Do we actually need to share the set across calls, or can get rid of this argument?

Vendor the compact-proof decoder from the released trie-db 0.31.0 verbatim into the test crate as a frozen snapshot of deployed decoder behavior, and assert that encodings produced by encode_compact_skip_duplicate_values decode with it into a hash-keyed database with every entry readable. This pins the backward compatibility the deduplicating encoder relies on, instead of leaving it as an argument in documentation.

encode_compact_skip_duplicates now emits each distinct trie node once, not just each detached value. A later occurrence of an already-emitted subtree keeps a plain hash reference (like any reference outside the partial trie), so the encoding never grows. The decoder threads a known-items map and re-inserts skipped subtrees at every position, reconstructing the same database as an un-deduplicated encoding.

skunert · 2026-07-13T10:34:44Z

You can do the de-duplication on the node level and not just on the value level.

We now also deduplicate nodes

bkchr · 2026-07-14T09:31:14Z

+					// `omit_children` bit stays unset, keeping a plain hash reference. The root is
+					// never skipped, so each encoding stays individually decodable when
+					// `seen_hashes` is threaded across successive encodings.
+					if !stack.is_empty() && seen_hashes.contains(node_hash.as_ref()) {


I don't get why you check here if stack is not empty?

This is needed to decide whether we are at the root of a child trie for example. If the root would be in seen_hashes, the encoding for the child trie would be empty.

bkchr · 2026-07-14T09:36:24Z

+		},
+		_ => 0,
+	};
+	for index in 0..entry.children.len() {


Why are you not directly iterating the entry.children?

Because we use it to access the bitmask

bkchr · 2026-07-14T09:36:49Z

+		if entry.hash_ref_children & (1u16 << index) == 0 {
+			continue
+		}


Suggested change

if entry.hash_ref_children & (1u16 << index) == 0 {

continue

}

This is the exact same check as below?

See comment here

bkchr · 2026-07-14T09:37:19Z

 	children: Vec<Option<ChildReference<C::HashOut>>>,
+	/// Bit mask of children kept as plain hash references, which may point at subtrees
+	/// deduplicated into an earlier occurrence (see [`encode_compact_skip_duplicates`]).
+	hash_ref_children: u16,


Suggested change

hash_ref_children: u16,

has_ref_children: bool,

And then just set it to true if there is one.

This is a bitmask that marks which children came on the wire as hash. We insert the deduplicated subtrees towards the end and iterate all the children. But at that point it is not possible to distinguish between items that came as detached value and items that where always Hash.

The ones that came as detached value where already inserted into the db, so we should not insert them again to keep the ref count in the db correct.

I am not getting that, we skip only dettached value? not duplicated trie node hash (could make sense though)? so would only need to look at the single value hash?

bkchr · 2026-07-14T09:52:59Z

+			if hash.is_none() {
+				// The node's detached value may have been deduplicated into an earlier occurrence
+				// (see `encode_compact_skip_duplicates`); re-insert it here too so `db` matches an
+				// un-deduplicated encoding. A miss means the value is not in the encoding, which is


If the hash is none and the the value is missing, it means the db is missing values and we should return an error?

What does value missing mean for you here?

If the hash is none that means that there was no detached node in the compact encoding. So we just have the hash.

If that hash belongs to a node that was deduplicated and we have seen it earlier, we need to reinsert to maintain refcount.

if the hash belongs to a node that we have not yet seen, that means the node is just not referenced in the proof, which should also be fine.

bkchr · 2026-07-14T09:53:26Z

+pub fn decode_compact_from_iter_with_known_items<'a, L, DB, I>(
+	db: &mut DB,
+	encoded: I,
+	known_items: &mut BTreeMap<Vec<u8>, DBValue>,


Do we actually need to supply this?

https://github.com/paritytech/trie/pull/233/changes#r3581312671

bkchr · 2026-07-14T09:54:16Z

+	DB: HashDB<L::Hash, DBValue>,
+	I: IntoIterator<Item = &'a [u8]>,
+{
+	decode_compact_from_iter_with_known_items::<L, DB, I>(db, encoded, &mut BTreeMap::new())


This method is also doing the same reconstruction of the database with the same counts. So, there is no real difference between these two?

The difference is just backwards compat. The previous method existed before and does not break anything. with_known_items is only needed if you want to pass through the known_items, which we could do in the scenario outlined here.

skunert · 2026-07-15T12:43:38Z

@cheme Would be super nice if you could support us with a review here :)

cheme · 2026-07-15T21:43:04Z

@cheme Would be super nice if you could support us with a review here :)

Sure, will look at it (may not be able to do tomorrow, but will try to do before next week).

cheme

Using seen_hashes make sense to me, I think it is good (should even be extended node hashes).
What I am not too sure, is why we try to use prefixedmemorydb or have proper rc count.

cheme · 2026-07-16T05:55:58Z

 }

+/// A value above every tested layout threshold, stored as a shared, hash-addressed value node.
+const SHARED_VALUE: &[u8] = &[4; 32];


Would just switch to 33 to have more case with value nodes (but no case with inline value).

cheme · 2026-07-16T06:20:17Z

+	let value_hash = &node_data[hash_plan.clone()];
+	if let Some(seen_hashes) = &seen_hashes {
+		if seen_hashes.contains(value_hash) {
+			return None


Suggested change

return None

// Do not reinsert duplicated value node

return None

Or any comment, just to show this is the line doing the work

cheme · 2026-07-16T06:24:47Z

+				let Some(node_hash) = node_hash else { continue };
+
+				if let Some(seen_hashes) = seen_hashes.as_deref_mut() {
+					// A subtree whose root was already emitted is skipped entirely; the parent's


Suggested change

// A subtree whose root was already emitted is skipped entirely; the parent's

// Avoid skipping entirely a subtree whose root was already emitted; the parent's

cheme · 2026-07-16T06:30:12Z

 	children: Vec<Option<ChildReference<C::HashOut>>>,
+	/// Bit mask of children kept as plain hash references, which may point at subtrees
+	/// deduplicated into an earlier occurrence (see [`encode_compact_skip_duplicates`]).
+	hash_ref_children: u16,


I am not getting that, we skip only dettached value? not duplicated trie node hash (could make sense though)? so would only need to look at the single value hash?

cheme · 2026-07-17T21:26:18Z

+/// counts) requires a re-inserting decoder, i.e. [`decode_compact_from_iter_with_known_items`].
+///
+/// Assumes occurrences of an item are interchangeable, as they are when `db` is hash-keyed. A
+/// prefixed `db` populating a duplicated subtree only below a later occurrence would drop it.


I am not sure I understand the sentence about Prefixed Db.

From what I remember a prefixed memory db should only be used with a trie to produce/record the payload to send to rocksdb. So generally I would think things would get easier if we avoid using it at all when doing a compact proof.
I may have missed a case (I remember seeing some fix passing, maybe about it), but right now I could not figure it to be usefull.

It seems in the PR, managing prefixed db is the main source of complexity. For instance, for subtree, if not requiring prefixed db, one can just record the first subtree and skip the other ones.

Similarily, Iirc, proofs do not need to have proper rc (rc of memory db being usefull only for the product/record the payload to send to paritydb). Execution against a state machine never change the original state (delta of a removed value is simply stored in a keyvalue change map, so the backend would not change until receiving the payload).

cheme · 2026-07-17T21:29:06Z

+					// never skipped, so each encoding stays individually decodable when
+					// `seen_hashes` is threaded across successive encodings. Sound only under the
+					// fixed-backing-set precondition (see `encode_compact_skip_duplicates`): the
+					// subtree below a seen hash must not have grown since it was emitted.


I am not too sure this is needed, generally proof is a record of state accesses, so would only need to be an unprefixed memorydb with any rc being fine.

At the trie level it may make sense still, but subtree/childtree is a substrate concept, it is odd to see it in the trie crate.

cheme · 2026-07-17T21:33:17Z

+/// would have inserted below this node. Hashes missing from `known_items` reference items outside
+/// the encoding and are skipped, like the holes an un-deduplicated encoding would leave.
+///
+/// `prefix` is `entry`'s node prefix; it is restored before returning.


IIUC the point of this function is to reinsert a subtree/childtree , which somehow would make sense to me to deduplicate over a different prefix, or account for proper Rc. (but as mentioned before I am unsure it make sense for a proof).

if needed, I would certainly feel like this should be part of substrate (yet I remember in the past having to put some substrate related design in the trie_codec, so there may be a reason here, but I don t see it immediatly).

cheme · 2026-07-17T21:38:10Z

+				// The node's detached value may have been deduplicated into an earlier occurrence
+				// (see `encode_compact_skip_duplicates`); re-insert it here too so `db` matches an
+				// un-deduplicated encoding. A miss means the value is not in the encoding, which is
+				// legal.


I see the point of reinserting, yet we are no sure the value was actually accessed here.

I would just not reinsert, as rc does not matter to me. If rc matters (not sure why), then we can insert indeed, but since we don’t know if it was skipped due to duplication or value unaccessed, we may obtain a wrong rc.

One solution might be to simply have another placeholder value for duplicated value, but then we are not really compatible with old proofs.

I wonder if we shall force proof to run on a unprefixed memorydb that do not allow insert and removal, so Rc or prefix would not matter.

lexnv

LGTM! Had a look over comments and addresses them in #234

Maybe one extra thing before merging would be 855ad48

skunert and others added 5 commits July 7, 2026 16:41

trie-db: count attached values in decode_compact's used-items result

37afef5

A trailing attached value node was not counted in the returned used-item count, so decoding concatenated encodings at that offset re-read the value bytes as a node.

Shorten comments

75728c1

skunert requested a review from bkchr July 7, 2026 15:12

bkchr reviewed Jul 9, 2026

View reviewed changes

skunert added 3 commits July 13, 2026 12:25

Shorten comment

29d7b10

bkchr reviewed Jul 14, 2026

View reviewed changes

Comment thread trie-db/src/iterator.rs Outdated

bkchr reviewed Jul 14, 2026

View reviewed changes

skunert added 2 commits July 14, 2026 18:56

Nits

f5362f1

Rename stack empty check to is_root

84b84bd

skunert requested a review from cheme July 15, 2026 12:42

skunert added 5 commits July 16, 2026 10:27

Improve fuzzing scenarios

669eaa6

Add better guided fuzzer

cbf5607

Change refcount invariant

be170a7

Fix compiler warning

9d829a8

Improve fuzzer

64d012c

cheme reviewed Jul 17, 2026

View reviewed changes

lexnv mentioned this pull request Jul 31, 2026

trie-db: Avoid cloning the hashes on the hot path #234

Open

lexnv approved these changes Jul 31, 2026

View reviewed changes

	if entry.hash_ref_children & (1u16 << index) == 0 {
	continue
	}

	return None
	// Do not reinsert duplicated value node
	return None

	// A subtree whose root was already emitted is skipped entirely; the parent's
	// Avoid skipping entirely a subtree whose root was already emitted; the parent's

Uh oh!

Conversation

skunert commented Jul 7, 2026

Summary

Compatibility

Uh oh!

bkchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skunert commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skunert commented Jul 15, 2026

Uh oh!

cheme commented Jul 15, 2026

Uh oh!

cheme left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexnv left a comment

Choose a reason for hiding this comment

Uh oh!