Skip to content

Feature/flexible persistence via ListOptions#136

Open
Grant McCloskey (MushuEE) wants to merge 2 commits into
agent-substrate:mainfrom
MushuEE:feature/flexible-persistence
Open

Feature/flexible persistence via ListOptions#136
Grant McCloskey (MushuEE) wants to merge 2 commits into
agent-substrate:mainfrom
MushuEE:feature/flexible-persistence

Conversation

@MushuEE
Copy link
Copy Markdown
Collaborator

Fixes #135

Decouples the control plane persistence layer (store.Interface) from rigid listing constraints by introducing a generic, non-ossifying query filtering surface.

Problem

Currently, the persistence listing methods (ListWorkers and ListActors) are all-or-nothing. To filter workers by pool or namespace, the scheduler must fetch all records from the database and perform in-memory filtering in Go. This approach:

  1. Scales poorly at high worker counts.
  2. Coupled the storage API surface tightly to structural changes.
  3. Prevents pluggable databases (like SQL or DynamoDB) from leveraging B-Tree indices to execute $O(1)$ query-time filtering.

Solution

  1. Introduced store.ListOptions: Added a generic ListOptions struct with a FieldSelector map[string]string parameter to the storage Interface listing methods.
  2. Non-Ossifying Mapping Strategy: Implemented allocation-free, fast switch-based field mapping helpers (matchesWorker, matchesActor) in the ateredis package. This lets the storage engine filter items internally before returning them.
  3. Scheduler Integration: Updated the API service handlers and scheduling workflow (AssignWorkerStep) to pass the options cleanly.
  4. Validation: Added robust unit tests verifying exact-match field selector filtering (e.g., by pool, empty actor_id to claim ready workers, and actor status).

Architectural Advantages

  • Decoupled API Surface: Adding new worker fields (e.g., node_name, gpu_type) does not require changing the store.Interface signatures.
  • Database Pluggability: Relational database drivers (like PostgreSQL or Spanner) can translate the FieldSelector directly into indexed SQL WHERE clauses.
  • Locality-Aware Ready-Checks: Allows the scheduler to query for free workers with specific snapshot locality hints (warm-cache lookups) with a clean fallback to the general pool.

@MushuEE Grant McCloskey (MushuEE) changed the title Feature/flexible persistence Feature/flexible persistence via ListOptions Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC: Flexible & Non-Ossifying Control Plane Persistence Layer

1 participant