Skip to content

feat(marketplace): tunnel pools, the allocator, and a node's data plane - #364

Merged
v0l merged 3 commits into
masterfrom
feat/marketplace-tunnel-alloc
Aug 6, 2026
Merged

feat(marketplace): tunnel pools, the allocator, and a node's data plane#364
v0l merged 3 commits into
masterfrom
feat/marketplace-tunnel-alloc

Conversation

@v0l

@v0l v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Increment 4a of work/marketplace.md. An approved marketplace node can now ask for the WireGuard tunnel that carries its guests' traffic back to an LNVPS route server.

Why this is 4a and not 4

Increment 4 as written was XL — a new addressing schema, an allocator, live route-server I/O, drift reconciliation, real Linux networking on somebody else's machine, and an end-to-end health gate. Split into:

  • 4a (this PR) — addressing and allocation
  • 4b — realise the peer on the route server, route the guest prefixes, reconcile tunnel against router_tunnel
  • 4c — the node's own wg0/br-lnvps, anti-spoof rules, MTU/MSS clamp, and the health gate that finally enables the host

Where addresses come from

tunnel recorded what was assigned to whom, but nothing said what there was to assign from: router carries no region, no endpoint, no server key and no address block. tunnel_pool supplies all four and is the tunnel equivalent of ip_range — same shape, same job, and allocate_subnet carves both. It is deliberately not columns on router: a route server can terminate several WireGuard interfaces, and a peer belongs to an interface, not to a machine.

The node generates its own keypair and presents the public half. That is why there is no WireGuard column on marketplace_node — the key belongs on the tunnel row, which does not exist until the node asks, and uk_tunnel_peer_pubkey then makes it unique fleet-wide for free.

Endpoints

Method Path Auth
POST/GET /api/v1/node/tunnel node token
GET/POST /api/admin/v1/tunnel_pools router::view/create
GET/PATCH/DELETE /api/admin/v1/tunnel_pools/{id} router::view/update/delete

Things that had to be built rather than assumed

  • The tunnel's router cannot drift from its pool's. tunnel (pool_id, router_id) references tunnel_pool (id, router_id), so the database rejects a tunnel claiming a pool on another machine rather than leaving it to whichever code path happens to check. Verified against a real MariaDB, including that a NULL pool_id skips the constraint — which is the pool-less case that keeps tunnel.router_id meaningful for a hand-configured peering.
  • The peer's address is the network address with its low bit set, not plus one. On a /31 or /127 that bit is always clear, so there is no overflow case and no arithmetic error to invent: the last link in a block is still a link.
  • A dual-stack pool's capacity is the smaller block's. A link of each family is handed out together, so reporting the roomier block would promise capacity that cannot be allocated.
  • Pools are tried in order, first with room wins. A second pool in a region exists because the first filled up or is being migrated away from; spreading nodes across both would leave neither drainable.
  • A block cannot be shrunk or removed under a live allocation, or that tunnel sits outside its own pool and the allocator hands its addresses to somebody else. A pool cannot be moved between route servers at all — so router_id is absent from the update request.
  • A node presenting a new key is re-pinned in place. A machine restored from backup would otherwise be unreachable for good; the addresses do not move, so nothing downstream is re-plumbed.

Allocation also fills in the blank control address that approval leaves on the backing host — the node's control API is reachable only through the tunnel. The host stays disabled: an allocation is paperwork, not a working tunnel.

Verification

  • Migration applied against a real MariaDB; the composite FK and the "a pool must have a block" CHECK were both exercised (drift rejected, block-less pool rejected, pool-less tunnel accepted)
  • cargo test --workspace --exclude lnvps_e2e -- --test-threads=1 — green
  • cargo llvm-cov: 100% line coverage on provisioner/tunnel.rs (16 tests) and 100% function + line coverage on admin/tunnel_pools.rs (13 tests)
  • Known gap, same as increment 3a: the two node-facing axum handlers are thin wrappers with no test, because lnvps_api's RouterState has no test harness. Their bodies are covered through the allocator; the admin pool handlers are covered end to end. Recorded in the work file.
  • ADMIN_API_ENDPOINTS.md, API_CHANGELOG.md and work/marketplace.md updated

v0l added 2 commits August 6, 2026 13:57
Increment 4a. An approved marketplace node can now ask for the WireGuard tunnel
that carries its guests' traffic back to an LNVPS route server.

Increment 4 as written was XL — a new addressing schema, an allocator, live
route-server I/O, drift reconciliation, real Linux networking on somebody else's
machine and an end-to-end health gate — so it is split into 4a (this),
4b (realise the peer on the route server, reconcile drift) and 4c (the node's
own networking and the health gate that finally enables the host).

`tunnel` recorded what was assigned to whom, but nothing said what there was to
assign *from*: `router` carries no region, no endpoint, no server key and no
address block, so an allocator had nothing to pick or carve. `tunnel_pool`
supplies all four and is the tunnel equivalent of `ip_range` — same shape, same
job, and `allocate_subnet` carves both. It is not a set of columns on `router`,
because a route server can terminate several WireGuard interfaces and a peer
belongs to an interface, not to a machine.

The node generates its own keypair and presents the public half; the private
half never leaves the operator's machine. That is why there is no WireGuard
column on `marketplace_node`: the key belongs on the `tunnel` row, which does
not exist until the node asks, and `uk_tunnel_peer_pubkey` then makes it unique
fleet-wide for free.

Things that had to be built rather than assumed:

- **The tunnel's router cannot drift from its pool's.** `tunnel (pool_id,
  router_id)` references `tunnel_pool (id, router_id)`, so the database rejects
  a tunnel claiming a pool on another machine instead of leaving it to whichever
  code path happens to check. Verified against a real MariaDB, including that a
  NULL `pool_id` skips the constraint — which is the pool-less case that keeps
  `tunnel.router_id` meaningful for a hand-configured peering.
- **The peer's address is the network address with its low bit set**, not plus
  one. On a /31 or /127 that bit is always clear, so there is no overflow case
  and no arithmetic error to invent: the last link in a block is still a link.
- **A dual-stack pool's capacity is the smaller block's.** A link of each family
  is handed out together, so reporting the roomier block would promise capacity
  that cannot be allocated.
- **Pools are tried in order, first with room wins.** A second pool in a region
  exists because the first filled up or is being migrated away from; spreading
  nodes across both would leave neither drainable.
- **A block cannot be shrunk or removed under a live allocation**, or that
  tunnel sits outside its own pool and the allocator hands its addresses to
  somebody else. A pool cannot be moved between route servers at all — every
  tunnel carved from it would point at an interface that is not there — so
  `router_id` is absent from the update request.
- **A node presenting a new key is re-pinned in place.** A machine restored from
  backup would otherwise be unreachable for good; the addresses do not move, so
  nothing downstream is re-plumbed.

Allocation also fills in the blank control address that approval leaves on the
backing host: the node's control API is reachable only through the tunnel. The
host stays disabled — an allocation is paperwork, not a working tunnel.
A pool recorded a public key an admin had pasted in, which describes an
interface somebody already built by hand. It could never create one, could not
rebuild it after the route server was reinstalled, and made standing up a new
route server a manual job with a database row bolted on afterwards.

LNVPS now owns the interface end to end:

- **The keypair is generated** on pool creation and the private key is stored
  encrypted, like every other credential in this schema. An existing interface
  can still be adopted by handing over its private key — the public half is
  always *derived*, never accepted, so a pool cannot be stored holding a pair
  that disagrees with itself, and the sync refuses to configure one that does.
- **The interface is pushed to the route server** over the existing
  `TunnelRouter`: created on pool creation, re-applied on edit, removed on
  delete, and re-drivable on demand through
  `POST /api/admin/v1/tunnel_pools/{id}/sync` — because the push fails for
  reasons that have nothing to do with the pool (a rebuilt route server, a
  rotated SSH key, a box that is down) and the fix is to try again, not to edit
  the row.
- **The listening socket is stated in full**, because a route server carries
  several interfaces. `listen_addr` + `listen_port` replace the single
  `endpoint` string, and what a peer dials is derived from them (IPv6
  bracketed) so the two cannot disagree. A WireGuard interface listens on
  *every* local address at its port, so the port — not the address — is what
  two interfaces on one machine collide over: `uk_tunnel_pool_router_port`
  enforces that, verified against a real MariaDB.

Two things the implementation had to get right rather than assume:

- **Re-applying an interface recreates it and drops every peer with it**, so it
  only happens when the key or port has actually drifted. A pool that merely
  changed name must not cut every node on it; enable/disable goes through
  `set_tunnel_enabled` instead.
- **A dead job queue must not lose the pool.** Create stores the row and logs
  the failed push, because returning an error for a successful write invites an
  admin to create it a second time; delete removes the row and logs loudly that
  an interface was left configured, because a pool nobody can delete while
  Redis is down would be worse.

Re-keying is deliberately explicit: it cuts every node holding the old public
key until it re-reads its configuration, so it happens only when `private_key`
is present in the request, never as a side effect of renaming a pool.
@v0l

v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a second commit reworking pool management after review: LNVPS now configures the interface, rather than recording one an admin built by hand.

  • Key material is generated and owned by LNVPS. tunnel_pool.private_key (encrypted at rest, like router.token) plus the derived public key. An existing interface can still be adopted by supplying its private key; the public half is always derived, never accepted, so a pool cannot hold a pair that disagrees with itself — and the sync refuses to push one that does.
  • The interface is pushed to the route server through the existing TunnelRouter: created on pool creation, re-applied on edit, removed on delete, plus POST /api/admin/v1/tunnel_pools/{id}/sync to re-drive it (the push fails for reasons unrelated to the pool — rebuilt route server, rotated SSH key, box down — and the fix is to retry, not to edit the row).
  • The listening socket is stated in full, since a route server carries several interfaces: listen_addr + listen_port replace the single endpoint string, and what a peer dials is derived from them with IPv6 bracketed. A WireGuard interface listens on every local address at its port, so the port is what two interfaces collide over — uk_tunnel_pool_router_port enforces it, verified against a real MariaDB.

Two behaviours worth flagging in review:

  • Re-applying recreates the interface and drops its peers, so it only happens when the key or port has actually drifted. Renaming a pool must not cut every node on it; enable/disable goes through set_tunnel_enabled.
  • A dead job queue does not lose the pool. Create stores the row and logs the failed push (returning an error for a successful write invites a duplicate); delete removes the row and logs loudly that an interface was left configured.

Re-keying is explicit — private_key present in a PATCH — because it cuts every node holding the old key until it re-reads its config.

New tests: worker sync_tunnel_pool (creates the interface with the right key/port, second sync is a no-op, mismatched keypair refused), remove_tunnel_interface (removes it, idempotent, clears the cache), keypair generation/derivation, socket validation and uniqueness, adoption, explicit re-keying, and the dead-queue paths. Whole workspace suite green; admin/tunnel_pools.rs at 100% function coverage for production code.

A pool carried an admin-typed `interface` column. That is a name for something
LNVPS creates, on a machine that is not LNVPS-exclusive, which makes it three
problems at once:

- a route server carries interfaces nobody here configured, and nothing stopped
  a pool being pointed at one of them — the next sync would rewrite somebody
  else's tunnel with our key;
- two pools could be given the same name by an admin typing it twice;
- the name could be edited afterwards, which silently orphans the interface the
  pool actually created and configures a second one.

The interface is now `wgln<id>`, derived from the pool's own id. The prefix
keeps a managed interface distinguishable from everything else on the box, ids
are unique so names cannot collide, and there is nothing to edit. The column,
its unique key, and the create/update fields are all gone; `interface` is still
*returned*, because an admin looking at a route server needs to know which
interface belongs to which pool.

`uk_tunnel_pool_router_port` is now the only per-router uniqueness constraint,
which is the one that matters: a WireGuard interface listens on every local
address at its port, so the port is what two interfaces on a machine collide
over.

Adoption of an existing interface still works, and is now honest about what it
adopts: the key material, not the name. The interface LNVPS configures is the
managed one either way.
@v0l

v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Third commit: the interface name is now derived from the pool id as wgln<id>, and the interface column is gone.

An admin-typed name for something LNVPS creates, on a machine that is not LNVPS-exclusive, was three problems at once:

  • a route server carries interfaces nobody here configured, and nothing stopped a pool being pointed at one of them — the next sync would rewrite somebody else's tunnel with our key;
  • two pools could be given the same name by an admin typing it twice;
  • the name could be edited afterwards, silently orphaning the interface the pool actually created and configuring a second one.

The prefix keeps a managed interface distinguishable from everything else on the box, ids are unique so names cannot collide, and there is nothing left to edit. interface is still returned — an admin looking at a route server needs to know which interface belongs to which pool — but never accepted.

uk_tunnel_pool_router_interface is dropped; uk_tunnel_pool_router_port is now the only per-router uniqueness constraint, which is the one that actually matters (an interface listens on every local address at its port, so the port is what two of them collide over).

Adoption of an existing interface still works and is now honest about what it adopts: the key material, not the name.

Migration re-verified against a real MariaDB; workspace suite green.

@v0l
v0l merged commit 08725fe into master Aug 6, 2026
11 checks passed
@v0l
v0l deleted the feat/marketplace-tunnel-alloc branch August 6, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant