feat(internity): introduce string interning crate

[geeknoid]

Add compact single-threaded and concurrent string interners with frozen, lock-free readers, no-std and Serde support, property tests, Loom models, and Criterion, Gungraun, and memory benchmarks.

Lookup benchmarks use 60,000 distinct identifier-like strings via INTERNITY_BENCH_CORPUS_SIZE=60000; memory benchmarks use the default 6,000 strings. Lower is better.

Lookup implementation	Time
internity (live)	234.85 us
internity (frozen)	214.25 us
lasso	236.60 us
string-interner	309.74 us
symbol_table	574.28 us
string_cache	269.66 us

Memory implementation	Handle	Filled	Lookup
internity Lexicon	4 B	172.0 KiB	96.5 KiB
internity ThreadedLexicon	4 B	181.1 KiB	98.8 KiB
lasso	4 B	264.2 KiB	224.2 KiB
string-interner	4 B	204.0 KiB	204.0 KiB
symbol_table	4 B	241.3 KiB	241.3 KiB
string_cache	8 B	351.7 KiB	351.7 KiB

[Copilot]

Pull request overview

Introduces a new internity crate providing compact string interning with both a single-threaded (Lexicon) and concurrent (ThreadedLexicon) implementation, plus a frozen Reader for fast lock-free resolution. The PR also adds extensive tests, benchmarks, and performance documentation to support correctness and compare against other Rust interners.

Changes:

• Added internity crate (no-std + alloc baseline, optional std for concurrency, optional serde) with Sym handles and a sealed Reader abstraction.
• Added correctness coverage via integration tests, property tests (Bolero), and concurrency modeling (Loom).
• Added benchmarking + reporting tooling (Criterion, Gungraun/Callgrind, memory footprint harness) and performance/comparison docs.

Reviewed changes

Copilot reviewed 30 out of 33 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
README.md	Adds internity to the repo’s primary crates list.
CHANGELOG.md	Links the new internity crate changelog from the workspace changelog index.
Cargo.toml	Adds workspace dependency entries needed by internity (and competitors used in benches).
Cargo.lock	Records resolved dependencies for the new crate and benchmark competitors.
.spelling	Adds terminology used by internity docs/code (e.g., CSR, dedup, sharded).
crates/internity/Cargo.toml	Defines the new crate’s features, dependencies, benches, and Loom test target.
crates/internity/src/lib.rs	Crate root: public API exports, feature gating, and high-level documentation.
crates/internity/src/lexicon.rs	Implements the single-threaded interner with dense Sym encoding + freeze to Reader.
crates/internity/src/threaded_lexicon.rs	Implements the concurrent sharded interner and freeze-to-reader logic.
crates/internity/src/sym.rs	Defines Sym handle representation and encoding/decoding helpers.
crates/internity/src/reader.rs	Defines the sealed Reader trait for frozen, thread-safe resolution.
crates/internity/src/flat_reader.rs	Frozen reader implementation for Lexicon’s dense layout.
crates/internity/src/shard.rs	Shard wrapper using RwLock/upgradable-read for concurrent interning.
crates/internity/src/shard_write.rs	Shard write-state: dedup table + CSR offsets + byte buffer.
crates/internity/src/shard_reader.rs	Frozen per-shard storage and resolution routine.
crates/internity/src/sharded_reader.rs	Frozen reader implementation for ThreadedLexicon (shard + local decode).
crates/internity/src/storage.rs	Centralizes unchecked UTF-8 reconstruction for fast resolve paths.
crates/internity/src/symbol_map.rs	Adds Sym-keyed map/set support via a specialized hasher.
crates/internity/src/serde_impls.rs	Implements serde for Sym and interners (feature-gated).
crates/internity/tests/basic.rs	Broad integration tests covering API behavior, freezing, concurrency, serde, etc.
crates/internity/tests/bolero_internity.rs	Property tests for dedup/resolve/freezing and raw-handle range checks.
crates/internity/tests/loom_internity.rs	Loom model tests for concurrent dedup and snapshot behavior.
crates/internity/benches/compare.rs	Criterion wall-clock benchmark suite comparing multiple interners and thread counts.
crates/internity/benches/counts/main.rs	Gungraun/Callgrind harness entrypoint (Linux-only runner).
crates/internity/benches/counts/linux.rs	Deterministic instruction/cache-count benches for hot paths + competitors.
crates/internity/benches/mem.rs	Memory footprint benchmark using a tracking global allocator.
crates/internity/scripts/perf_report.rs	Script to run benches and regenerate docs/PERF.md.
crates/internity/docs/PERF.md	Generated performance report output committed to the repo.
crates/internity/docs/COMPARISON.md	Detailed comparative analysis across the Rust string interner ecosystem.
crates/internity/README.md	Crate README content for docs.rs / GitHub browsing.
crates/internity/CHANGELOG.md	Per-crate changelog for internity.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

✅ Version increments look sufficient

cargo semver-checks compared the 2 crate(s) this PR publishes against their previous version-bump commit in git history. Every version increment is sufficient for the detected API changes.

Crate	Baseline	Baseline commit	This PR	Minimum required	Status
internity	new crate	—	0.1.0	0.1.0	✅ ok
internity_macros	new crate	—	0.1.0	0.1.0	✅ ok

This check is informational and does not block the merge.

View the check run

[Copilot]

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 33 changed files in this pull request and generated no new comments.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (78aa447) to head (eb5cc4f).

Additional details and impacted files

@@            Coverage Diff            @@
##             main     #604     +/-   ##
=========================================
  Coverage   100.0%   100.0%             
=========================================
  Files         443      461     +18     
  Lines       42764    44553   +1789     
=========================================
+ Hits        42764    44553   +1789     

Flag	Coverage Δ
linux	100.0% <100.0%> (?)
linux-arm	100.0% <100.0%> (?)
windows	100.0% <100.0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

crates/internity/src/de/mod.rs:48

• The reference-style rustdoc links include ThreadedLexicon, which is not available without the std feature. This creates broken intra-doc links in serde-only (no_std) builds. After switching the earlier prose to non-linked ThreadedLexicon text, these reference links can be removed.

//! [`Sym`]: crate::Sym
//! [`Lexicon`]: crate::Lexicon
//! [`LocalLexicon`]: crate::LocalLexicon
//! [`ThreadedLexicon`]: crate::ThreadedLexicon

crates/internity/src/de/deserialize_in.rs:26

• The reference-style links for ThreadedLexicon/ThreadedLexicon::deserialize_in are broken when the std feature is off. Since the prose can avoid linking to ThreadedLexicon, these reference links should be removed to keep rustdoc clean in no_std builds.

/// [`Lexicon`]: crate::Lexicon
/// [`LocalLexicon`]: crate::LocalLexicon
/// [`LocalLexicon::deserialize_in`]: crate::LocalLexicon::deserialize_in
/// [`ThreadedLexicon`]: crate::ThreadedLexicon
/// [`ThreadedLexicon::deserialize_in`]: crate::ThreadedLexicon::deserialize_in

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 44 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated 8 comments.

Comments suppressed due to low confidence (2)

crates/internity/src/sym.rs:100

• The workspace lints enable clippy::unwrap_used, so this expect(...) triggers a warning (and CI denies warnings). Add an explicit #[expect(clippy::unwrap_used, reason = ...)] documenting that exceeding the u32 capacity is treated as a programming error / documented panic condition.
  crates/internity/src/local_lexicon.rs:203
• The workspace lints enable clippy::unwrap_used, so this expect(...) triggers a warning (and CI denies warnings). Add an explicit #[expect(clippy::unwrap_used, reason = ...)] on resolve to document that this is the intentional panicking variant.

    #[inline]
    #[must_use]
    pub fn resolve(&self, sym: Sym) -> &str {
        self.try_resolve(sym).expect("internity: Sym does not belong to this interner")

[Copilot]

Pull request overview

Copilot reviewed 45 out of 46 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 47 out of 48 changed files in this pull request and generated no new comments.

[ralfbiedert]

Second review pass at 98484156ba6ba250258c8b8b5c16bb9b2460dd0c.

The previous custom-hasher serialization issue and duplicated-corpus insertion benchmark are fixed. Remaining/new findings:

1. DeserializeIn does not honor Serde field-schema attributes. crates/internity_macros/src/lib.rs:44,64-101,179-189,220-225 ignores serde(rename), rename_all, default, skip, and flatten, so ordinary Deserialize and DeserializeIn can accept different wire formats. Either support the relevant attributes or reject/document the restricted schema.
2. Shared freeze remains a non-point-in-time snapshot (threaded_lexicon.rs:165-178,300-302; shard.rs:80-86). A handle can be returned by an insertion before freeze returns yet be absent from the reader.
3. ThreadedLexicon serialization still copies the complete corpus because serde_impls.rs:105-110 clones and freezes while the original reference remains alive.
4. Same-shard dedup hits remain serialized through the single upgradable-read slot at shard.rs:54-64.
5. The concurrent Criterion timers at benches/compare.rs:382-445 include barrier wake-up and scheduling overhead instead of using the repository parallel benchmark harness.
6. The Loom model explicitly tests a different read/write/recheck algorithm than production's upgradable-lock transition.

Production unsafe remains isolated to storage.rs; the offset and UTF-8 invariants appear sound. All-feature tests, Loom, Miri, Clippy, rustfmt, CI, and facade docs succeeded.

[ralfbiedert]

Approved after the second-round review comment above.

[geeknoid]

Thanks for the thorough second pass. Addressed at 3bbd9180 (amended). Point-by-point:

1. Serde field-schema attributes — DeserializeIn now honors the deserialize-affecting Serde attributes so it accepts the same wire format as ordinary Deserialize. New internity_macros/src/attrs.rs parses both namespaces; the codegen supports container rename_all/rename/deny_unknown_fields/default/transparent/expecting and field rename (incl. the deserialize/serialize split), alias, default, skip/skip_deserializing, and with/deserialize_with. Unsupported representations (flatten, borrow, tagged/untagged, remote, from/try_from/into) are now rejected with a clear compile error rather than silently diverging. New tests/serde_attr_parity.rs asserts byte-for-byte acceptance parity against serde::Deserialize for each attribute. This mirrors the sibling multitude crate's approach.
2. Shared freeze point-in-time — build_reader now acquires read guards on all shards up front, copies, then releases, so an in-flight insertion cannot upgrade-to-write mid-snapshot; a handle observed before freeze returns is present in the reader. Steady-state hot path is unchanged; the sole-owner move path is untouched.
3. ThreadedLexicon serialization copy — serialization now streams strings directly in shard order under per-shard read guards via serialize_seq, dropping the clone().freeze() corpus copy. Wire format is unchanged.
4. Same-shard upgradable slot — left as-is by design. The single upgradable-read slot is what makes the same-shard miss→insert transition race-free without a second lock acquisition; serializing dedup misses within one shard is the intended trade-off, and cross-shard traffic (the common case) stays parallel.
5. Concurrent bench overhead — the harness now times only each worker's own loop (clock started after the start barrier), takes the max across workers per round, and drops construction/spawn/join/teardown from the measured region. I did not migrate to bench_on_threadpool() because that helper is only referenced in docs/benchmarks.md and isn't implemented in the repo, and many_cpus_benchmarking uses a processor-pair/NUMA model that doesn't fit internity's N-thread throughput sweep.
6. Loom model framing — the file is renamed to loom_algorithm_sketches.rs and the module/test docs now state explicitly that these are algorithm sketches exploring interleavings, not a model-check of the production upgradable-lock transition.

CI, Clippy, rustfmt, spellcheck, and doc2readme are green locally.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

[geeknoid]

@ralfbiedert — thanks again for the thorough second-round review. I've pushed changes (rebased onto latest main) addressing the findings:

1. DeserializeIn ignored serde field-schema attributes. The internity_macros derive now parses and honors the relevant #[serde(...)] schema attributes (rename, rename_all, alias, default/default = "...", skip/skip_deserializing, deny_unknown_fields, container default), so a type's Deserialize and DeserializeIn accept the same wire format. Unsupported cases (flatten, borrow, tagged/untagged/remote/from) are rejected at compile time rather than silently diverging. Added tests/serde_attr_parity.rs to lock the parity down.

2. freeze was not a point-in-time snapshot. build_reader now acquires read guards on all shards up front, before copying any blob. While every guard is held no intern can commit (a miss can't upgrade to the write lock while a reader is present), so the copied state is a single point-in-time snapshot instead of a per-shard-torn one. Guards are taken in index order and intern only ever locks one shard, so this can't deadlock. The sole-owner freeze(self) zero-copy move fast-path (Arc::try_unwrap) is retained.

3. Serializing ThreadedLexicon copied the whole corpus. Dropped the self.clone().freeze() snapshot. Serialization now streams strings directly in shard order under a per-shard read guard (try_for_each_str → serialize_element), holding one shard's lock at a time and allocating zero corpus copy. Emission order is unchanged, so deserialization still reproduces identical Sym handles.

4. Same-shard dedup hits on the single upgradable-read slot. Kept the current fill-oriented path per your guidance (the read-first/recheck variant regressed insertion, the crate's primary workload) and documented the tradeoff plus the "freeze before a read-heavy phase" guidance.

6. Loom tests framed as production model-checking. Renamed tests/loom_internity.rs → tests/loom_algorithm_sketches.rs and strengthened the module disclaimer to make explicit that these model an abstract read/drop/write-recheck algorithm — not production's parking_lot upgradable-guard atomic-upgrade transition (which Loom cannot instrument) — and must not be read as model-checking the production lock.

All tests pass across feature combinations and CI is green.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

crates/internity/docs/DESIGN.md:245

• The freeze documentation here says the shared/"snapshot" path walks shards one at a time and is not point-in-time, but the implementation holds read guards on all shards before copying any of them (see ThreadedLexiconInner::build_reader). This is a point-in-time snapshot; the doc should match the code to avoid misleading readers.

The snapshot path acquires read guards on **all** shards up front, before any
blob is copied. While every guard is held no concurrent `intern` can commit (a
miss cannot upgrade to the write lock while a reader is present), so the copied
state is a single **point-in-time snapshot** even when other clones are still

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

crates/internity_macros/src/lib.rs:531

• The generated Visitor::visit_seq for named structs reads the declared fields but never checks for trailing elements, so inputs like [field0, ..., fieldN, extra] are silently accepted. This diverges from serde::Deserialize behavior and undermines the trailing_sequence_fields_are_rejected integration test expectations. After reading the expected fields, probe for one extra IgnoredAny element and return invalid_length if present.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated no new comments.