Skip to content

feat(timeseries): add time-series analysis extension with embedding similarity and drift detection - #44

Open
chiangchenghsin-hash wants to merge 10 commits into
LadybugDB:mainfrom
chiangchenghsin-hash:feat/timeseries-extension
Open

feat(timeseries): add time-series analysis extension with embedding similarity and drift detection#44
chiangchenghsin-hash wants to merge 10 commits into
LadybugDB:mainfrom
chiangchenghsin-hash:feat/timeseries-extension

Conversation

@chiangchenghsin-hash

Copy link
Copy Markdown
Contributor

Summary

Adds a new timeseries extension with two table functions for time-series analysis on embedding vectors:

EMBEDDING_SIMILARITY

Compute cosine similarity and per-dimension feature similarity between two embedding vectors of any dimension (not hardcoded).

CALL embedding_similarity(
    CAST([0.1, 0.2, 0.3], 'DOUBLE[]'),
    CAST([0.1, 0.2, 0.4], 'DOUBLE[]')
) RETURN cosine_similarity, feature_similarity, dimension_count;

Returns: cosine_similarity ([-1, 1]), feature_similarity ([0, 1]), dimension_count.

DETECT_DRIFT_POINTS

Detect drift points in a sequence of embeddings by computing pairwise cosine distances between consecutive vectors. Generic labels (commit hashes, timestamps, version numbers) replace hardcoded chapter numbers.

CALL detect_drift_points(
    CAST([1.0, 0.0, 0.0, 0.0, 1.0, 0.0], 'DOUBLE[]'),
    2,
    CAST([100, 200], 'INT64[]'),
    3,
    threshold := 0.1
) RETURN label, drift_magnitude, significance, direction;

Returns: label (INT64), drift_magnitude (DOUBLE), significance (DOUBLE), direction ("up"/"down").

Use Cases

  • Architecture drift: embeddings of code snapshots per commit → detect when architecture changed significantly
  • Content drift: embeddings of document revisions over time → detect when content drifted
  • Behavior drift: embeddings of API response patterns → detect behavioral changes
  • Vector comparison: compare any two embeddings regardless of dimension

Design

  • Pure C++ implementation, no external dependencies (no Arrow, no OpenMP)
  • Uses LadybugDB's SimpleTableFunc pattern
  • inferInputTypes callback for ANY → LIST<DOUBLE> type resolution
  • Data prepared via Cypher COLLECT at app layer — no table scans needed
  • All algorithms are deterministic (no randomness, no LLM calls)

Files

File Purpose
timeseries/CMakeLists.txt Top-level CMake
timeseries/src/function/CMakeLists.txt Function library
timeseries/src/main/CMakeLists.txt Extension library
timeseries/src/include/function/timeseries_function.h Function registration structs
timeseries/src/include/main/timeseries_extension.h Extension header
timeseries/src/main/timeseries_extension.cpp Extension loader
timeseries/src/function/embedding_similarity.cpp EMBEDDING_SIMILARITY implementation
timeseries/src/function/detect_drift_points.cpp DETECT_DRIFT_POINTS implementation
timeseries/test/test_files/timeseries.test E2E tests (3 cases)
timeseries/README.md API documentation

Motivation

This extension was developed for architecture governance (detecting when code architecture drifts from the intended design via embedding comparison). It generalizes domain-specific bitemporal analysis functions into reusable primitives that work with any embedding space.

- EMBEDDING_SIMILARITY: generic N-dimension cosine/feature similarity
- DETECT_DRIFT_POINTS: generic drift detection on sequential embeddings
  (generalized from bitemporal's character_similarity + detect_turning_points)
…ne similarity

Generalized from character_similarity (hardcoded 4 features) to accept
embeddings of any dimension. Returns cosine similarity + mean feature
similarity + dimension count.
…n embeddings

Generalized from detect_turning_points (hardcoded chapter labels) to accept
generic labels (commit hashes, timestamps, version numbers, etc.).
Use cases: architecture drift, content drift, behavior drift detection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant