Scale startup for 10k+ streams via lazy stream indexes, startup manifest bootstrap, and bounded index FDs

[Copilot]

Startup was dominated by eager stream-index opening and full filesystem scans, which does not scale with large stream counts and can exhaust file descriptors. This change decouples stream discovery from index opening, adds a persisted startup manifest fast-path, and enforces LRU limits for open secondary index handles.

• Lazy stream registration and open-on-first-use

  • EventStore.registerStream() now records stream descriptors instead of opening each stream index during startup.
  • Stream indexes are opened only when actually needed (e.g., version/read/query/commit paths), while _all remains eagerly available.
  • Closed-stream transitions keep read behavior intact while aligning with lazy descriptor flow.

• Startup manifest fast-path (startupState)

  • Added StartupState with versioned/checksummed payloads and atomic writes.
  • ReadableStorage.open() now attempts bootstrap from a clean manifest and falls back to full scan when state is missing/dirty/inconsistent (including primary-length mismatch).
  • Added dirty/clean lifecycle hooks around layout mutations and state persistence after reconciliation.

• Generic file-handle pooling + index FD limits

  • Introduced reusable FileHandlePool.
  • Refactored PartitionPool onto the shared abstraction.
  • Added IndexPool and maxOpenIndexes to cap concurrently open secondary index files with LRU eviction.

• State-sync integration for correctness-sensitive flows

  • Wired state invalidation/persistence through index/partition lifecycle operations, including stream create/delete/close and storage repair/truncate/reindex paths.
  • Preserves optimistic concurrency/query behavior under lazy index opening.

• Targeted behavior coverage

  • Added specs for:
    • no eager stream-index opening on startup,
    • startup manifest fast-path + dirty fallback,
    • bounded secondary index handles via maxOpenIndexes.

// Example: startup-state + bounded index handles
const store = new EventStore({
  storageDirectory,
  storageConfig: {
    startupState: { enabled: true },
    maxOpenIndexes: 256
  }
});

[albe]

@copilot I dislike the heavy spread of those startupState calls, for two reasons:

• they always come in two and hence need to be both placed correctly to not break the semantics. They do not directly belong to the logic currently executed, but rather are orthogonal side effects that should be executed at some specific points
• every invocation writes a potentially pretty large JSON file to disk, twice, which very likely outweighs the other step currently taken (e.g. create new partition), i.e. it will trash performance of common paths

Under what circumstance is the dirty mark strictly necessary? Can the state persistence only be invoked when new indexes/partitions are created? Can we avoid the primaryIndexLength check, hence needing to update the state on every document write?

[Copilot]

Addressed in 3093427: removed broad startup-state dirty/save pairs from hot paths, introduced withStartupStateMutation(...) for structural changes, and removed the primaryLength check dependency from startup-state bootstrapping.