Skip to content

Add support for dcb.events style DCB query - #331

Merged
albe merged 15 commits into
mainfrom
dcb-query
Jul 18, 2026
Merged

Add support for dcb.events style DCB query#331
albe merged 15 commits into
mainfrom
dcb-query

Conversation

@albe

@albe albe commented Jul 15, 2026

Copy link
Copy Markdown
Owner

This pull request introduces several major improvements to EventStore's querying capabilities, focusing on DCB (Dynamic Consistency Boundary) support, dynamic stream indexing, and enhanced documentation. The most important changes include a new generic streamSources mechanism for automatic stream indexing, the addition of tagsAccessor for tag-based streams, a new DCB query shorthand syntax, and comprehensive updates to the documentation explaining these features and their trade-offs.

Dynamic Stream Indexing and Query Enhancements

  • Added the streamSources configuration option, allowing users to define custom stream indexes for any payload property. This enables automatic maintenance of dedicated streams for properties like tenant IDs, types, or tags, with O(1) index routing. (src/EventStore.js [1] [2]
  • Introduced the tagsAccessor option as a shorthand for creating tag-based streams from an array property in the payload. This supports efficient tag queries and is integrated with DCB query resolution. (src/EventStore.js [1] [2]
  • Implemented DCB query shorthand syntax: store.query({ items: [{ types, tags }] }) can now be used instead of manually constructing the nested selector algebra. This requires typeAccessor and/or tagsAccessor to be configured. (src/EventStore.js [1] [2] [3]

Documentation Updates

  • Expanded and clarified the DCB documentation to explain the new query model, the role of typeAccessor and tagsAccessor, and the difference between matcher-only and tag-stream queries. Includes practical examples, trade-off discussions, and advanced selector algebra details. (docs/dcb.md [1] [2] [3] [4] [5] [6] [7]
  • Added a section to the advanced documentation describing dynamic stream indexing and how to use the new streamSources, typeAccessor, and tagsAccessor options. (docs/advanced.md docs/advanced.mdR63-R95)
  • Updated the changelog to reflect all new features, including the DCB query shorthand, tagsAccessor, and streamSources. Also noted a minor bugfix in JoinStream. (docs/changelog.md docs/changelog.mdR7-R13)

These changes make it much easier to configure and use DCB-style queries, improve performance for high-cardinality or high-throughput use cases, and provide clear guidance on when to use matcher-only versus stream-indexed queries.

albe added 6 commits July 14, 2026 23:19
This allows tagsAccessor to use fast object matching and indexing lookup.
This allows to specify generic stream indexing properties in the options, which the store will make sure are indexed in dedicated streams.
| path                 | getPropertyAtPath | compileAccessor | factor |
|----------------------|------------------:|----------------:|-------:|
| 'type'               |          84 ns    |        12 ns    |   ~7×  |
| 'meta.kind'          |         120 ns    |        41 ns    |   ~3×  |
| 'meta.nested.value'  |         150 ns    |        65 ns    |  ~2.3× |
@coveralls

coveralls commented Jul 15, 2026

Copy link
Copy Markdown

Coverage Status

coverage: 98.288% (+0.04%) from 98.252% — dcb-query into main

Comment thread src/utils/dcbUtil.js Outdated
Comment thread src/utils/dcbUtil.js Outdated
Comment thread src/utils/metadataUtil.js Outdated
Copilot finished work on behalf of albe July 15, 2026 18:29
@albe

albe commented Jul 16, 2026

Copy link
Copy Markdown
Owner Author

@copilot the new matcher semantic for includes needs to be added to the bufferMatcher as well. However, since the input value is not distinctive from exact single property matching, this would lead to this casing needing to do alternative checks, which in turn makes that (most common) case much slower.
Hence I propose a change like this:
Instead of matcher: { prop: scalarValue } for array includes use { prop: { $has: scalarValue} }, i.e. a new matcher operator.
Then for the bufferMatcher in that case, build a "key":[ prefix and a node matcher function of indexOfSameLevel(serializedScalarValue, currentPosition+prefix.length) >= 0. A scalar matcher property stays as only an exact scalar match.

Add an implementation for that and adjust the documentation accordingly, mentioning the new $has operator and explain how to use that in DCB when not using tagsAccessor streams as an alternative that is more performant than a matcher function.

Copilot AI commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@copilot the new matcher semantic for includes needs to be added to the bufferMatcher as well. However, since the input value is not distinctive from exact single property matching, th...

Done. Summary of changes:

  • Reverted the auto-containment behavior in propertyMatchesValue — scalar matchers against array document values no longer perform an implicit Array.includes.
  • Added a new $has matcher operator:
    • Object mode: Array.isArray(value) && value.includes(expected) via the compiled operator checks.
    • Raw buffer mode: lone $has compiles to a "key":[ prefix + indexOfSameLevel(serializedScalar, valueStart) !== -1 — the array-element scan skips nested objects/arrays and stops at the array's closing ]. String scalars route through this fast path directly (their serialized form starts with ", matching indexOfSameLevel's validation).
  • Wired tagsAccessor (and array-valued streamSources entries, auto-detected in ensureStreams via Array.isArray(raw)) through a new buildTagMatcherFn that emits { payload: { <path>: { $has: value } } } matchers.
  • Extended IndexMatcher.findDiscriminant to recognize lone $has matchers and treat the $has scalar as the discriminant value, preserving the O(1) discriminant-table route for tag-stream matchers.
  • Documented $has in docs/streams.md and docs/api.md; added a "prefer $has over function matcher" section in docs/dcb.md under matcher-only queries; added a changelog entry.
  • Test coverage: object-mode $has (positive, negative, non-array, absent-key), raw-mode $has (containment, wrong-depth rejection, combined with other properties), and updated IndexMatcher tests to use the new $has matcher shape.

All 663 tests pass; CodeQL is clean.

Copilot finished work on behalf of albe July 16, 2026 16:52
Comment thread docs/dcb.md
Comment thread docs/streams.md Outdated
Comment thread src/utils/metadataUtil.js Outdated
Comment thread src/utils/metadataUtil.js Outdated
Comment thread src/EventStore.js Outdated
Copilot finished work on behalf of albe July 16, 2026 17:50

Copilot AI commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

@albe I've opened a new pull request, #335, to work on those changes. Once the pull request is ready, I'll request review from you.

Comment thread src/EventStore.js Outdated
Copilot finished work on behalf of albe July 18, 2026 11:43
@albe

albe commented Jul 18, 2026

Copy link
Copy Markdown
Owner Author

@copilot improve code coverage of all newly added/changed code in this PR

Copilot AI commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

@copilot improve code coverage of all newly added/changed code in this PR

Added tests covering the previously-uncovered branches in newly-added PR code:

  • util.compileAccessor: null/undefined inputs for single-part paths, and nested paths with missing/non-object intermediate segments.
  • IndexMatcher.isLoneHasScalar (via findDiscriminant): non-$has operator matchers, matchers with multiple operator keys, empty-array discriminant values, and arrays containing non-scalar elements — all correctly falling through to unclassifiedMatchers.

Coverage on the changed source files:

  • src/utils/util.js: branches 93.33 % → 96.87 %
  • src/IndexMatcher.js: branches 95.71 % → 98.66 %
  • src/utils/dcbUtil.js: 100 % (unchanged)
  • src/utils/metadataUtil.js: statements/lines 99.25 %, branches 99.34 %

Remaining uncovered lines in the changed files are all pre-existing code from main (verified via git blame) — no newly-added PR statements or branches are uncovered.

Copilot finished work on behalf of albe July 18, 2026 11:58
@albe
albe merged commit 6c849cb into main Jul 18, 2026
13 checks passed
@albe
albe deleted the dcb-query branch July 18, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants