Roadmap questions

[spunkedy]

@markovejnovic we talked on Reddit.

I figured having a diagram and thoughts here might be a little easier to communicate some of the ideas. I was thinking through ways of possibly separating out some of the layers to make them portable as an interface to be able to do things like:

• create extensions like vfkit / apple containers for the runtime
• scale out without having to need postgres as a hard dependency ( thinking through auto scaling groups etc )

If you are open to something like this, I can flush it out much more. In a PR

Proposal: split hyper_control / hyper_agent, behind a pluggable registry interface

Problem

:hyper is one Mix package today. Every node — whether it's fielding gRPC calls and
scheduling, or just running Firecracker VMs — runs identical code and links ecto_sql +
postgrex with live DB credentials, because agent-side code calls straight into
Hyper.Img.Db.* (Ecto/Postgres) rather than through an interface.

Concretely, three call sites do this today:

Call site	What it does	File
Hyper.Node.Img.Server.resolve_layers/1 → Db.Image.resolve_chain/1	VM boot: img_id → ordered layer/blob chain	lib/hyper/node/img/server.ex
Hyper.Node.Img.with_image_lease/3 → Db.Lease.bump/3, Db.Lease.release/1	Per-VM lease heartbeat while a VM lives	lib/hyper/node/img.ex
Hyper.Metering.Usage.record/1	Periodic CPU-usage row insert, called from FireVMM.Meter	lib/hyper/metering/usage.ex

Everything else that's Postgres-hardwired (Hyper.Img.create/2, create_derived/3, all of
Hyper.Img.Db.Gc) is write/ingest or singleton-GC work that belongs to the control side
regardless of any split.

Notably, the equivalent problem on the bytes side is already solved:
Hyper.Node.Layer.Repo (lib/hyper/node/layer/repo.ex) only ever resolves
layer_id -> path, and its moduledoc already says it may be "backed by anything you like:
a plain filesystem, an NFS drive." There's no Ecto in it. The gap is purely on the
metadata side.

Proposed split

Three Mix packages:

• hyper_control — scheduler, gRPC API, Postgres (Hyper.Img.Db.Repo + migrations),
  GC. 1–3 nodes, HA quorum.
• hyper_agent — VM execution: Node.VMSupervisor/FireVMM, Budget, Reaper,
  Users, the setuid suidhelper. N nodes.
• hyper_proto — shared contracts: Vm.Id/Vm.Spec, Unit.*, generated gRPC +
  Firecracker bindings. No Ecto, no scheduler, no Postgres.

Control and agents stay joined on the same Distributed Erlang mesh they use today —
Horde's CRDT gossip (Hyper.Cluster.Routing, Hyper.Cluster.Budget) keeps working
unmodified; Hyper.Cluster.Gc moves into hyper_control since it needs Postgres.

Module map

Module / area	Today	Proposed package	Port / adapter
Hyper.Img.Db.Repo + migrations	lib/hyper/img/db/repo.ex, priv/repo/	hyper_control	—
Hyper.Cluster.Scheduler	lib/hyper/cluster/scheduler.ex	hyper_control	—
Hyper.Img, Hyper.Img.Db.Gc	lib/hyper/img.ex, lib/hyper/img/db/gc*.ex	hyper_control	—
Hyper.Grpc.Server (external API)	lib/hyper/grpc/*.ex	hyper_control	—
Hyper.Node.Img / Img.Server — chain resolve, lease bump/release	lib/hyper/node/img.ex, img/server.ex	hyper_agent	Hyper.Img.Registry — Postgres or Remote adapter
Hyper.Metering.Usage (called from FireVMM.Meter)	lib/hyper/metering/usage.ex	hyper_agent	Hyper.Img.Registry.record_usage/1
Hyper.Node.Layer.Repo — blob path lookup	lib/hyper/node/layer/repo.ex	hyper_agent	Layer.Repo — local/NFS or blob-proxy adapter
Hyper.Node.* — VMSupervisor, FireVMM, Users, Budget, Reaper	lib/hyper/node/**	hyper_agent	—
suidhelper, guest-agent	native/**	hyper_agent	—
Hyper.Cluster.Routing / Budget (CRDTs)	lib/hyper/cluster/{routing,budget}.ex	started by both	—
Vm.Id/Vm.Spec, Unit.*, generated bindings	lib/hyper/vm/{id,spec}.ex, lib/unit/**, proto/**	hyper_proto	—

The interface layer

Two ports, each with a direct (colocated) adapter and a remote (funneled through
hyper_control) adapter — chosen independently of each other:

defmodule Hyper.Img.Registry do
  @moduledoc "Port: agent-side access to image lineage and lease/usage state."

  @callback resolve_chain(Hyper.Img.id()) :: [layer()]
  @callback lease_bump(Hyper.Img.id(), Hyper.Vm.Id.t(), Unit.Time.t()) ::
              {:ok, term()} | {:error, term()}
  @callback lease_release(Hyper.Vm.Id.t()) :: :ok
  @callback record_usage(Hyper.Metering.Usage.attrs()) :: :ok | {:error, term()}
end

Port	Adapter	Backing	Extra hop?	Best for
Hyper.Img.Registry (new)	Registry.Postgres	This node's own Ecto pool — Db.Image, Db.Lease, Metering.Usage	No	Trusted subnet — today's behavior, unchanged
Hyper.Img.Registry (new)	Registry.Remote	Call to hyper_control over the existing libcluster mesh	Yes — RTT on VM boot + lease heartbeat	NAT'd / cross-region agents with no direct DB reachability
Hyper.Node.Layer.Repo (exists today)	local / NFS mount	File.stat/1 against Cfg.Dirs.layer_dir()	No	Already shipped — works as-is
Hyper.Node.Layer.Repo (exists today)	blob-store proxy	Streamed / range-fetched from hyper_control or an object store	Yes — per-layer fetch on first mount	Cross-region deploys with no shared POSIX filesystem

Both remote adapters can start as plain Distributed Erlang calls, since agents already join
the same cluster as hyper_control for Horde's CRDTs — no new wire protocol needed until an
agent has to leave that mesh entirely, at which point the call becomes a gRPC RPC on
hyper.proto instead.

Trade-offs

Pros

• Execution nodes never ship Postgres drivers or DB credentials — smaller trust surface per VM host.
• Scheduler, GC, and the gRPC API redeploy independently of hundreds of live-VM agents.
• Bounded connection-pool pressure: control nodes' pools, not N-agents × pool_size.
• Reaches topologies today's design can't: NAT'd, cross-region, or edge agents with no DB line-of-sight.
• Agent-side tests run against a fake port — no Postgres needed in CI for that code path.

Cons

• New latency + failure mode on VM boot: resolve_chain becomes a network call, not a local query.
• Lease heartbeat now depends on control-plane reachability, adding a partition-vs-DB-down failure mode to classify.
• Control plane must be its own HA target, or it's a new bottleneck — today "HA" is just "Postgres is HA."
• Loses free transactionality: Ecto.Multi writes need a purpose-built transactional RPC when remote.
• More to operate: an internal RPC surface with its own versioning, auth boundary, and retry policy.

[spunkedy]

Current

Rough idea

[markovejnovic]

This is almost completely AI generated. Could you kindly spend some time rewriting this proposal by hand? I do not want to spend my human time reviewing AI proposals.

Thanks

[markovejnovic]

For future reference, I plan on keeping the roadmap here Hyper Roadmap.

[spunkedy]

Yes it is AI written as I asked it to summarize a couple implementations that I thought through.

I don't mind refining it completely, but I don't want to go through and invest the 2 hours towards something if you want to keep some of the features coupled.

To be clear, help is only helpful if it helps. If this isn't where you want to take it, out of respect. I don't want to waste your time or mine.

Do you like ideas of:

• a pluggable interface/adapter pattern that allows for other implementations like broadway, broadway_sqs ecto etc.
  • firecfracker
  • kvm
  • apple virtualization
  • bhyve
• a decoupling of hard requirement of postgres where it becomes optional for the executor nodes
  • runner nodes that can be decoupled from being in vpc/region etc

If so I don't mind drawing up some implementation plans / PRs off of some of those ideas to get alignment.