feat(marketplace): tunnel pools, the allocator, and a node's data plane - #364
Conversation
Increment 4a. An approved marketplace node can now ask for the WireGuard tunnel that carries its guests' traffic back to an LNVPS route server. Increment 4 as written was XL — a new addressing schema, an allocator, live route-server I/O, drift reconciliation, real Linux networking on somebody else's machine and an end-to-end health gate — so it is split into 4a (this), 4b (realise the peer on the route server, reconcile drift) and 4c (the node's own networking and the health gate that finally enables the host). `tunnel` recorded what was assigned to whom, but nothing said what there was to assign *from*: `router` carries no region, no endpoint, no server key and no address block, so an allocator had nothing to pick or carve. `tunnel_pool` supplies all four and is the tunnel equivalent of `ip_range` — same shape, same job, and `allocate_subnet` carves both. It is not a set of columns on `router`, because a route server can terminate several WireGuard interfaces and a peer belongs to an interface, not to a machine. The node generates its own keypair and presents the public half; the private half never leaves the operator's machine. That is why there is no WireGuard column on `marketplace_node`: the key belongs on the `tunnel` row, which does not exist until the node asks, and `uk_tunnel_peer_pubkey` then makes it unique fleet-wide for free. Things that had to be built rather than assumed: - **The tunnel's router cannot drift from its pool's.** `tunnel (pool_id, router_id)` references `tunnel_pool (id, router_id)`, so the database rejects a tunnel claiming a pool on another machine instead of leaving it to whichever code path happens to check. Verified against a real MariaDB, including that a NULL `pool_id` skips the constraint — which is the pool-less case that keeps `tunnel.router_id` meaningful for a hand-configured peering. - **The peer's address is the network address with its low bit set**, not plus one. On a /31 or /127 that bit is always clear, so there is no overflow case and no arithmetic error to invent: the last link in a block is still a link. - **A dual-stack pool's capacity is the smaller block's.** A link of each family is handed out together, so reporting the roomier block would promise capacity that cannot be allocated. - **Pools are tried in order, first with room wins.** A second pool in a region exists because the first filled up or is being migrated away from; spreading nodes across both would leave neither drainable. - **A block cannot be shrunk or removed under a live allocation**, or that tunnel sits outside its own pool and the allocator hands its addresses to somebody else. A pool cannot be moved between route servers at all — every tunnel carved from it would point at an interface that is not there — so `router_id` is absent from the update request. - **A node presenting a new key is re-pinned in place.** A machine restored from backup would otherwise be unreachable for good; the addresses do not move, so nothing downstream is re-plumbed. Allocation also fills in the blank control address that approval leaves on the backing host: the node's control API is reachable only through the tunnel. The host stays disabled — an allocation is paperwork, not a working tunnel.
A pool recorded a public key an admin had pasted in, which describes an
interface somebody already built by hand. It could never create one, could not
rebuild it after the route server was reinstalled, and made standing up a new
route server a manual job with a database row bolted on afterwards.
LNVPS now owns the interface end to end:
- **The keypair is generated** on pool creation and the private key is stored
encrypted, like every other credential in this schema. An existing interface
can still be adopted by handing over its private key — the public half is
always *derived*, never accepted, so a pool cannot be stored holding a pair
that disagrees with itself, and the sync refuses to configure one that does.
- **The interface is pushed to the route server** over the existing
`TunnelRouter`: created on pool creation, re-applied on edit, removed on
delete, and re-drivable on demand through
`POST /api/admin/v1/tunnel_pools/{id}/sync` — because the push fails for
reasons that have nothing to do with the pool (a rebuilt route server, a
rotated SSH key, a box that is down) and the fix is to try again, not to edit
the row.
- **The listening socket is stated in full**, because a route server carries
several interfaces. `listen_addr` + `listen_port` replace the single
`endpoint` string, and what a peer dials is derived from them (IPv6
bracketed) so the two cannot disagree. A WireGuard interface listens on
*every* local address at its port, so the port — not the address — is what
two interfaces on one machine collide over: `uk_tunnel_pool_router_port`
enforces that, verified against a real MariaDB.
Two things the implementation had to get right rather than assume:
- **Re-applying an interface recreates it and drops every peer with it**, so it
only happens when the key or port has actually drifted. A pool that merely
changed name must not cut every node on it; enable/disable goes through
`set_tunnel_enabled` instead.
- **A dead job queue must not lose the pool.** Create stores the row and logs
the failed push, because returning an error for a successful write invites an
admin to create it a second time; delete removes the row and logs loudly that
an interface was left configured, because a pool nobody can delete while
Redis is down would be worse.
Re-keying is deliberately explicit: it cuts every node holding the old public
key until it re-reads its configuration, so it happens only when `private_key`
is present in the request, never as a side effect of renaming a pool.
|
Pushed a second commit reworking pool management after review: LNVPS now configures the interface, rather than recording one an admin built by hand.
Two behaviours worth flagging in review:
Re-keying is explicit — New tests: worker |
A pool carried an admin-typed `interface` column. That is a name for something LNVPS creates, on a machine that is not LNVPS-exclusive, which makes it three problems at once: - a route server carries interfaces nobody here configured, and nothing stopped a pool being pointed at one of them — the next sync would rewrite somebody else's tunnel with our key; - two pools could be given the same name by an admin typing it twice; - the name could be edited afterwards, which silently orphans the interface the pool actually created and configures a second one. The interface is now `wgln<id>`, derived from the pool's own id. The prefix keeps a managed interface distinguishable from everything else on the box, ids are unique so names cannot collide, and there is nothing to edit. The column, its unique key, and the create/update fields are all gone; `interface` is still *returned*, because an admin looking at a route server needs to know which interface belongs to which pool. `uk_tunnel_pool_router_port` is now the only per-router uniqueness constraint, which is the one that matters: a WireGuard interface listens on every local address at its port, so the port is what two interfaces on a machine collide over. Adoption of an existing interface still works, and is now honest about what it adopts: the key material, not the name. The interface LNVPS configures is the managed one either way.
|
Third commit: the interface name is now derived from the pool id as An admin-typed name for something LNVPS creates, on a machine that is not LNVPS-exclusive, was three problems at once:
The prefix keeps a managed interface distinguishable from everything else on the box, ids are unique so names cannot collide, and there is nothing left to edit.
Adoption of an existing interface still works and is now honest about what it adopts: the key material, not the name. Migration re-verified against a real MariaDB; workspace suite green. |
Increment 4a of
work/marketplace.md. An approved marketplace node can now ask for the WireGuard tunnel that carries its guests' traffic back to an LNVPS route server.Why this is 4a and not 4
Increment 4 as written was XL — a new addressing schema, an allocator, live route-server I/O, drift reconciliation, real Linux networking on somebody else's machine, and an end-to-end health gate. Split into:
tunnelagainstrouter_tunnelwg0/br-lnvps, anti-spoof rules, MTU/MSS clamp, and the health gate that finally enables the hostWhere addresses come from
tunnelrecorded what was assigned to whom, but nothing said what there was to assign from:routercarries no region, no endpoint, no server key and no address block.tunnel_poolsupplies all four and is the tunnel equivalent ofip_range— same shape, same job, andallocate_subnetcarves both. It is deliberately not columns onrouter: a route server can terminate several WireGuard interfaces, and a peer belongs to an interface, not to a machine.The node generates its own keypair and presents the public half. That is why there is no WireGuard column on
marketplace_node— the key belongs on thetunnelrow, which does not exist until the node asks, anduk_tunnel_peer_pubkeythen makes it unique fleet-wide for free.Endpoints
POST/GET/api/v1/node/tunnelGET/POST/api/admin/v1/tunnel_poolsrouter::view/createGET/PATCH/DELETE/api/admin/v1/tunnel_pools/{id}router::view/update/deleteThings that had to be built rather than assumed
tunnel (pool_id, router_id)referencestunnel_pool (id, router_id), so the database rejects a tunnel claiming a pool on another machine rather than leaving it to whichever code path happens to check. Verified against a real MariaDB, including that a NULLpool_idskips the constraint — which is the pool-less case that keepstunnel.router_idmeaningful for a hand-configured peering.router_idis absent from the update request.Allocation also fills in the blank control address that approval leaves on the backing host — the node's control API is reachable only through the tunnel. The host stays disabled: an allocation is paperwork, not a working tunnel.
Verification
cargo test --workspace --exclude lnvps_e2e -- --test-threads=1— greencargo llvm-cov: 100% line coverage onprovisioner/tunnel.rs(16 tests) and 100% function + line coverage onadmin/tunnel_pools.rs(13 tests)lnvps_api'sRouterStatehas no test harness. Their bodies are covered through the allocator; the admin pool handlers are covered end to end. Recorded in the work file.ADMIN_API_ENDPOINTS.md,API_CHANGELOG.mdandwork/marketplace.mdupdated