diff --git a/ADMIN_API_ENDPOINTS.md b/ADMIN_API_ENDPOINTS.md index 892ea3dc..86e7645f 100644 --- a/ADMIN_API_ENDPOINTS.md +++ b/ADMIN_API_ENDPOINTS.md @@ -3898,7 +3898,7 @@ Body: Where tunnel inner addresses are allocated from — the tunnel equivalent of an `ip_range`. A pool names the route server that terminates the peers, the region whose nodes may use it, the socket the data plane listens on, and the blocks -that point-to-point links are carved out of. +that node addresses are carved out of. **LNVPS manages the interface.** Creating a pool generates its WireGuard keypair, stores the private key encrypted, and configures the interface on the @@ -3928,10 +3928,20 @@ Required Permission: `router::view` Query Parameters: `limit` (default 50, max 100), `offset`, `region_id`. Returns a paginated list of `AdminTunnelPoolInfo`. `links_used` is how many -links are already carved out; `links_total` is what the **smaller** of the two -blocks can supply, because a dual-stack pool hands out one link of each family -together — reporting the roomier block would promise capacity that cannot be -allocated. `endpoint` is derived from `listen_addr` and `listen_port` (IPv6 +nodes are already placed; `links_total` is what the **smaller** of the two +blocks can supply, because a dual-stack pool hands out one address of each +family together — reporting the roomier block would promise capacity that cannot +be allocated. + +A node takes a single address (`/32` or `/128`), not a link: WireGuard is layer +3 and point-to-point, so the node needs no gateway on its side and a +point-to-point link would spend two addresses describing something that needs +one. The route server holds **one** address for the whole pool, carrying the +block's own prefix so every node in it is on-link — an address per node would +put thousands of addresses on a single interface for a /16 pool. That block's +network address, the route server's address after it, and (on IPv4) its +broadcast address are reserved, which is why a /24 places 253 nodes rather than +256. `endpoint` is derived from `listen_addr` and `listen_port` (IPv6 bracketed) so what a peer is told to dial cannot disagree with the socket that was configured. @@ -4007,6 +4017,19 @@ or port has drifted is re-applied, and a pool whose stored keypair does not agree with itself is refused rather than configured into something no node could connect to. +The sync then **reconciles the pool's peers**: every tunnel allocated from the +pool is configured on the interface with its `AllowedIPs`, the route server gets +its own address on the pool, and each guest address is routed down +the interface. This always runs, and it matters most right after the interface +was created or re-applied — that is a fresh interface with no peers at all, and +every node on it is cut until they are put back. + +Peers are reconciled again on the routine router poll, so a peer wiped by a +reboot, a stale key nobody removed, or a guest address assigned since the last +push is corrected without an admin doing anything. A peer on an LNVPS-managed +interface that no allocation accounts for is **removed**: `wgln*` interfaces are +LNVPS's, so an unclaimed key there is either a revoked node or somebody else's. + ### Reports #### Time Series Report diff --git a/API_CHANGELOG.md b/API_CHANGELOG.md index f601ef0c..23886f5e 100644 --- a/API_CHANGELOG.md +++ b/API_CHANGELOG.md @@ -56,7 +56,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added -- **Marketplace nodes get a data plane** — an approved node can now ask for the WireGuard tunnel that carries its guests' traffic back to an LNVPS route server. `POST /api/v1/node/tunnel` (node token) presents the node's WireGuard **public** key and returns everything needed to bring the tunnel up: its own inner addresses, the route server's addresses on the same links, the server's public key, the `host:port` to dial, the keepalive and the MTU; `GET /api/v1/node/tunnel` reads the same back after a restart. The private key is generated on the node and never leaves it, which is why the key is presented rather than issued. Idempotent: retrying returns the allocation already made, and a node that presents a *new* key (restored from backup, state directory lost) is re-pinned in place without its addresses moving — refusing would leave a machine that can never be reached again. Addresses come from a new **tunnel pool**, administered at `GET/POST /api/admin/v1/tunnel_pools`, `GET/PATCH/DELETE /api/admin/v1/tunnel_pools/{id}` and `POST /api/admin/v1/tunnel_pools/{id}/sync` under the `router` resource: a route server, a region, the socket the data plane listens on (`listen_addr` + `listen_port`, unique per route server because a WireGuard interface listens on every local address at its port) and the v4/v6 blocks that /31 and /127 links are carved from. **LNVPS manages the interface end to end**: creating a pool generates its keypair, stores the private key encrypted and configures the interface on the route server; editing re-applies it; deleting removes it. The interface is named `wgln` from the pool's id and is not settable — a route server also carries interfaces LNVPS did not configure, and a fixed prefix keeps a managed one from being confused with, or clobbering, those. An admin never pastes key material for an interface they set up by hand, so standing up a new route server is an API call rather than a manual job with a database row bolted on afterwards — an existing interface can still be adopted by supplying its private key, and the public half is always derived, never accepted. The private key is never returned by any endpoint. Allocation also fills in the blank control address that node approval leaves on the backing host — the node's control API is only reachable through the tunnel. The host stays **disabled**: an allocation is paperwork, not a working tunnel, and the peer is not configured on the route server until the next increment. +- **A node takes one address, not a point-to-point link** — `address4`/`address6` in the tunnel response are now a `/32` and a `/128`, and `gateway4`/`gateway6` are one address shared by every node on the pool rather than a per-node link address. WireGuard is layer 3 and point-to-point: the node needs no gateway on its own side (`ip route add default dev wg0` suffices), so a `/31` spent two addresses describing something that needs one — and forced the route server to carry one address per node on a single interface, thousands of them on a /16 pool. Pool capacity is reported accordingly: a /24 places 253 nodes (256 less the block's network address, the route server's address after it, and the broadcast address) where it previously reported 128 links. + +- **A marketplace node's tunnel is now actually configured on the route server** — allocating a tunnel (`POST /api/v1/node/tunnel`) previously wrote down addresses and a key and stopped there; the node's peer is now pushed to the route server, so the tunnel carries traffic instead of only existing on paper. The peer's `AllowedIPs` is the node's own inner addresses plus exactly the guest addresses LNVPS assigned to it, which is also the anti-spoof boundary: WireGuard drops an inbound packet whose source is not on that list, so a node cannot source traffic as another node's customer. The route server takes an address on each point-to-point link, and each guest address is routed down the interface — `AllowedIPs` decides which *peer* a packet already headed for the tunnel belongs to, it does not put the packet there. Peers are also reconciled on the routine router poll, so an address assigned to a guest since the last push, a peer wiped by a route-server reboot, and a stale key nobody removed are all corrected without an admin doing anything; the reconcile reports what had drifted rather than quietly fixing it. The backing host stays **disabled** — a configured peer is still not a proven path, and the health gate that enables it comes with the node-side data plane. + +- **Marketplace nodes get a data plane** — an approved node can now ask for the WireGuard tunnel that carries its guests' traffic back to an LNVPS route server. `POST /api/v1/node/tunnel` (node token) presents the node's WireGuard **public** key and returns everything needed to bring the tunnel up: its own inner addresses (a `/32` and a `/128`), the route server's address on the pool, the server's public key, the `host:port` to dial, the keepalive and the MTU; `GET /api/v1/node/tunnel` reads the same back after a restart. The private key is generated on the node and never leaves it, which is why the key is presented rather than issued. Idempotent: retrying returns the allocation already made, and a node that presents a *new* key (restored from backup, state directory lost) is re-pinned in place without its addresses moving — refusing would leave a machine that can never be reached again. Addresses come from a new **tunnel pool**, administered at `GET/POST /api/admin/v1/tunnel_pools`, `GET/PATCH/DELETE /api/admin/v1/tunnel_pools/{id}` and `POST /api/admin/v1/tunnel_pools/{id}/sync` under the `router` resource: a route server, a region, the socket the data plane listens on (`listen_addr` + `listen_port`, unique per route server because a WireGuard interface listens on every local address at its port) and the v4/v6 blocks that node addresses are carved from. **LNVPS manages the interface end to end**: creating a pool generates its keypair, stores the private key encrypted and configures the interface on the route server; editing re-applies it; deleting removes it. The interface is named `wgln` from the pool's id and is not settable — a route server also carries interfaces LNVPS did not configure, and a fixed prefix keeps a managed one from being confused with, or clobbering, those. An admin never pastes key material for an interface they set up by hand, so standing up a new route server is an API call rather than a manual job with a database row bolted on afterwards — an existing interface can still be adopted by supplying its private key, and the public half is always derived, never accepted. The private key is never returned by any endpoint. Allocation also fills in the blank control address that node approval leaves on the backing host — the node's control API is only reachable through the tunnel. The host stays **disabled**: an allocation is paperwork, not a working tunnel, and the peer is not configured on the route server until the next increment. - **Admin approval for marketplace nodes** — operator-registered hardware can now be reviewed and admitted to the fleet. `GET /api/admin/v1/marketplace/nodes` (filterable by `status`/`operator_id`) is the review queue, `POST /api/admin/v1/marketplace/nodes/{id}/approve` admits a node, `PATCH /api/admin/v1/marketplace/nodes/{id}` suspends, drains or re-tiers it, and `DELETE /api/admin/v1/marketplace/nodes/{id}` rejects a registration outright — there is no `rejected` state, so an operator can fix what was wrong and register the same machine again. Approval is the **only** transition into `approved` and the only place the gates are checked: a node with no pinned TLS certificate cannot be reached and is refused, and where the region's company charges a listing fee, that fee must be paid *to that company* — a fee paid to a cheaper company does not list hardware elsewhere. Approving a node creates its backing host **disabled and with an empty control address**, because the node is reachable only over a data-plane tunnel that does not exist yet; capacity defaults to zero rather than being guessed, since real figures arrive with node telemetry. Suspending or draining also disables that host, so placement stops immediately instead of depending on the scheduler learning about node status. Two new permission resources, deliberately separate: `marketplace_node` (placement state) and `marketplace_operator` (revenue-share override and payout configuration, via `GET`/`PATCH /api/admin/v1/marketplace/operators[/{id}]`) — stopping a misbehaving node should not require the ability to change what someone is paid. diff --git a/lnvps_api/src/api/marketplace.rs b/lnvps_api/src/api/marketplace.rs index cd522048..ab971f5e 100644 --- a/lnvps_api/src/api/marketplace.rs +++ b/lnvps_api/src/api/marketplace.rs @@ -18,8 +18,8 @@ use serde::{Deserialize, Serialize}; use chrono::Utc; use lnvps_api_common::{ - ApiData, ApiError, ApiResult, NODE_TOKEN_TTL_SECS, Nip98Auth, NodeAuth, issue_node_token, - session_auth_enabled, + ApiData, ApiError, ApiResult, NODE_TOKEN_TTL_SECS, Nip98Auth, NodeAuth, WorkJob, + issue_node_token, session_auth_enabled, }; use lnvps_db::{ IntervalType, MarketplaceNode, MarketplaceNodeStatus, MarketplaceOperator, @@ -574,6 +574,25 @@ async fn v1_node_request_tunnel( let allocation = crate::provisioner::allocate_node_tunnel(&this.db, &auth.node, &key) .await .map_err(|e| ApiError::new(e.to_string()))?; + + // Realise the peer on the route server. Queued rather than awaited: the + // node has what it needs to configure its own end either way, and making + // the answer wait on an SSH round trip to a route server would fail the + // request for something the node cannot fix. A failure to queue is logged + // and left to the periodic reconcile, which pushes the same peer. + if let Err(e) = this + .work_sender + .send(WorkJob::SyncNodeTunnel { + tunnel_id: allocation.tunnel.id, + }) + .await + { + log::error!( + "Allocated tunnel {} for node {} but could not queue its peer push: {e}", + allocation.tunnel.id, + auth.node.id + ); + } ApiData::ok(allocation.into()) } @@ -1002,12 +1021,14 @@ mod tests { let allocation = crate::provisioner::NodeTunnel { tunnel: lnvps_db::Tunnel { id: 1, - address4: Some("10.66.0.1/31".to_string()), - address6: Some("fd00:66::1/127".to_string()), + address4: Some("10.66.0.2/32".to_string()), + address6: Some("fd00:66::2/128".to_string()), keepalive: Some(25), ..Default::default() }, pool: lnvps_db::TunnelPool { + cidr4: Some("10.66.0.0/24".to_string()), + cidr6: Some("fd00:66::/64".to_string()), public_key: vec![0x33; 32], listen_addr: "rs.example".to_string(), listen_port: 51820, @@ -1017,12 +1038,12 @@ mod tests { }; let api: ApiNodeTunnel = allocation.into(); - assert_eq!(api.address4.as_deref(), Some("10.66.0.1/31")); - assert_eq!(api.address6.as_deref(), Some("fd00:66::1/127")); - // The gateway is derived from the link rather than stored, so a second - // copy cannot disagree with the address it is paired with. - assert_eq!(api.gateway4.as_deref(), Some("10.66.0.0")); - assert_eq!(api.gateway6.as_deref(), Some("fd00:66::")); + assert_eq!(api.address4.as_deref(), Some("10.66.0.2/32")); + assert_eq!(api.address6.as_deref(), Some("fd00:66::2/128")); + // The gateway is derived from the pool's block rather than stored, and + // is the same one address for every node on the pool. + assert_eq!(api.gateway4.as_deref(), Some("10.66.0.1")); + assert_eq!(api.gateway6.as_deref(), Some("fd00:66::1")); assert_eq!(api.server_public_key, hex::encode([0x33; 32])); assert_eq!(api.endpoint, "rs.example:51820"); assert_eq!(api.keepalive, Some(25)); diff --git a/lnvps_api/src/mocks.rs b/lnvps_api/src/mocks.rs index fa67c80e..8ddadbee 100644 --- a/lnvps_api/src/mocks.rs +++ b/lnvps_api/src/mocks.rs @@ -52,6 +52,10 @@ pub struct MockRouter { tunnels: Arc>>, sessions: Arc>>, default_route: Arc>>, + /// Addresses configured per tunnel interface + addresses: Arc>>>, + /// Routes pointing down each tunnel interface + routes: Arc>>>, } impl Default for MockRouter { @@ -75,6 +79,10 @@ impl MockRouter { Arc::new(Mutex::new(HashMap::new())); static TL_SESSIONS: Arc>> = Arc::new(Mutex::new(HashMap::new())); + static TL_ADDRESSES: Arc>>> = + Arc::new(Mutex::new(HashMap::new())); + static TL_ROUTES: Arc>>> = + Arc::new(Mutex::new(HashMap::new())); static TL_DEFAULT_ROUTE: Arc>> = Arc::new(Mutex::new(Some(BgpRoute { prefix: "0.0.0.0/0".to_string(), @@ -87,6 +95,8 @@ impl MockRouter { tunnels: TL_TUNNELS.with(|t| t.clone()), sessions: TL_SESSIONS.with(|s| s.clone()), default_route: TL_DEFAULT_ROUTE.with(|d| d.clone()), + addresses: TL_ADDRESSES.with(|a| a.clone()), + routes: TL_ROUTES.with(|r| r.clone()), } } @@ -98,6 +108,31 @@ impl MockRouter { tunnels.clear(); let mut sessions = self.sessions.lock().await; sessions.clear(); + let mut addresses = self.addresses.lock().await; + addresses.clear(); + let mut routes = self.routes.lock().await; + routes.clear(); + } + + /// Addresses configured on a tunnel interface + pub async fn interface_addresses(&self, interface: &str) -> Vec { + let addresses = self.addresses.lock().await; + addresses.get(interface).cloned().unwrap_or_default() + } + + /// Routes pointing down a tunnel interface + pub async fn interface_routes(&self, interface: &str) -> Vec { + let routes = self.routes.lock().await; + routes.get(interface).cloned().unwrap_or_default() + } + + /// Peers configured on a tunnel interface + pub async fn peers(&self, interface: &str) -> Vec { + let tunnels = self.tunnels.lock().await; + match tunnels.get(interface).map(|t| &t.config) { + Some(crate::router::TunnelConfig::Wireguard(c)) => c.peers.clone(), + _ => vec![], + } } /// Seed a BGP session for tests @@ -257,6 +292,48 @@ impl TunnelRouter for MockRouter { Ok(()) } + async fn set_tunnel_peer( + &self, + interface: &str, + peer: &crate::router::WireguardPeer, + ) -> OpResult<()> { + let mut tunnels = self.tunnels.lock().await; + let Some(t) = tunnels.get_mut(interface) else { + return Err(OpError::Fatal(anyhow::anyhow!( + "No such tunnel interface: {interface}" + ))); + }; + if let crate::router::TunnelConfig::Wireguard(c) = &mut t.config { + // Keyed by public key, like the real thing: pushing the same peer + // twice updates it rather than duplicating it. + c.peers.retain(|p| p.public_key != peer.public_key); + c.peers.push(peer.clone()); + } + Ok(()) + } + + async fn remove_tunnel_peer(&self, interface: &str, public_key: &str) -> OpResult<()> { + let mut tunnels = self.tunnels.lock().await; + if let Some(t) = tunnels.get_mut(interface) + && let crate::router::TunnelConfig::Wireguard(c) = &mut t.config + { + c.peers.retain(|p| p.public_key != public_key); + } + Ok(()) + } + + async fn sync_tunnel_addresses(&self, interface: &str, addresses: &[String]) -> OpResult<()> { + let mut map = self.addresses.lock().await; + map.insert(interface.to_string(), addresses.to_vec()); + Ok(()) + } + + async fn sync_tunnel_routes(&self, interface: &str, prefixes: &[String]) -> OpResult<()> { + let mut map = self.routes.lock().await; + map.insert(interface.to_string(), prefixes.to_vec()); + Ok(()) + } + async fn tunnel_traffic(&self) -> OpResult> { let tunnels = self.tunnels.lock().await; Ok(tunnels diff --git a/lnvps_api/src/provisioner/tunnel.rs b/lnvps_api/src/provisioner/tunnel.rs index 308c439e..46fdec88 100644 --- a/lnvps_api/src/provisioner/tunnel.rs +++ b/lnvps_api/src/provisioner/tunnel.rs @@ -21,6 +21,7 @@ use ipnetwork::IpNetwork; use lnvps_db::{LNVpsDb, MarketplaceNode, RouterTunnelKind, Tunnel, TunnelPool}; use crate::provisioner::allocate_subnet; +use crate::router::WireguardPeer; /// A node's tunnel and the pool it came from. /// @@ -34,54 +35,64 @@ pub struct NodeTunnel { } impl NodeTunnel { - /// The route server's address on the inner link. + /// The route server's address, which every node on the pool shares. /// - /// Derived from the node's own address rather than stored: on a /31 (or - /// /127) there are exactly two addresses, LNVPS takes the first and the - /// node the second. A stored copy would be a second answer to a question - /// that already has one. + /// Derived from the pool's block rather than stored, and one address for + /// the whole pool rather than one per node: WireGuard is layer 3 and + /// point-to-point, with no ARP and no on-link requirement, so a per-node + /// link address bought nothing and cost the route server one address per + /// node on a single interface. pub fn gateway4(&self) -> Option { - link_gateway(self.tunnel.address4.as_deref()) + server_address(self.pool.cidr4.as_deref()).map(|a| bare_address(&a)) } - /// The route server's address on the inner IPv6 link. + /// The route server's IPv6 address on the pool. pub fn gateway6(&self) -> Option { - link_gateway(self.tunnel.address6.as_deref()) + server_address(self.pool.cidr6.as_deref()).map(|a| bare_address(&a)) } } -/// The first address of the link `addr` sits on, as plain text. -fn link_gateway(addr: Option<&str>) -> Option { - let net: IpNetwork = addr?.parse().ok()?; - Some(net.network().to_string()) -} - -/// The second address of `link`, which is the peer's side. +/// The route server's own address in `cidr`, as CIDR carrying the block's +/// prefix (`10.66.0.1/16`). /// -/// A /31 or /127 has no network or broadcast address to skip — that is the -/// point of using them for point-to-point links — so the peer's side is simply -/// the other of the two. +/// The first usable address of the block, so it is fixed for the life of the +/// pool: it is handed to every node as their gateway, and a value that moved +/// when the block was edited would strand all of them at once. /// -/// Setting the low bit rather than adding one is deliberate: on a link that -/// wide the network address always has it clear, so this cannot overflow and -/// there is no arithmetic failure case to invent an error for. -fn peer_address(link: &IpNetwork) -> String { - match link { +/// Carries the block's prefix rather than a host prefix so the route server +/// treats the whole pool as on-link — that is what makes one address serve +/// every node. +pub fn server_address(cidr: Option<&str>) -> Option { + let net: IpNetwork = cidr?.parse().ok()?; + let first = next_address(&net)?; + Some(format!("{first}/{}", net.prefix())) +} + +/// The address one above the network address of `net`. +fn next_address(net: &IpNetwork) -> Option { + Some(match net { IpNetwork::V4(v4) => { - let addr = std::net::Ipv4Addr::from(u32::from(v4.network()) | 1); - format!("{}/{}", addr, v4.prefix()) + std::net::Ipv4Addr::from(u32::from(v4.network()).checked_add(1)?).into() } IpNetwork::V6(v6) => { - let addr = std::net::Ipv6Addr::from(u128::from(v6.network()) | 1); - format!("{}/{}", addr, v6.prefix()) + std::net::Ipv6Addr::from(u128::from(v6.network()).checked_add(1)?).into() } - } + }) +} + +/// `10.66.0.1/16` -> `10.66.0.1`. +fn bare_address(cidr: &str) -> String { + cidr.split_once('/').map_or(cidr, |(a, _)| a).to_string() } -/// Point-to-point prefix lengths. A /31 (RFC 3021) and a /127 (RFC 6164) are -/// exactly two addresses, so a link costs nothing beyond the two ends. -const LINK_PREFIX_V4: u8 = 31; -const LINK_PREFIX_V6: u8 = 127; +/// A node holds a single address, not a link. +/// +/// WireGuard needs no gateway on the node's side (`ip route add default dev +/// wg0` is enough on a point-to-point layer 3 interface), so a /31 spent two +/// addresses to describe something that needs one — and forced the route server +/// to carry an address per node. +const NODE_PREFIX_V4: u8 = 32; +const NODE_PREFIX_V6: u8 = 128; /// Fetch a node's existing data plane, if it has one. pub async fn get_node_tunnel( @@ -252,7 +263,7 @@ fn peer_name(node: &MarketplaceNode) -> String { } /// Carve the next free link out of `pool`, returning `(address4, address6)` in -/// the peer's own form (`10.0.0.1/31`). +/// the peer's own form (`10.0.0.2/32`). /// /// A pool with both blocks must supply both halves: a node given only one /// family would silently be single-stack, which is the kind of thing that is @@ -267,23 +278,43 @@ async fn carve_link( .iter() .flat_map(|t| [t.address4.clone(), t.address6.clone()]) .flatten() - // A stored address is the peer's own (`10.0.0.1/31`); parsing keeps the - // host bits, and the overlap check works off the network address, so - // the whole link is excluded rather than just the one address. .filter_map(|a| a.parse::().ok()) .collect(); let address4 = match pool.cidr4.as_deref() { - Some(cidr) => Some(carve_one(cidr, LINK_PREFIX_V4, &taken, pool)?), + Some(cidr) => Some(carve_one(cidr, NODE_PREFIX_V4, &taken, pool)?), None => None, }; let address6 = match pool.cidr6.as_deref() { - Some(cidr) => Some(carve_one(cidr, LINK_PREFIX_V6, &taken, pool)?), + Some(cidr) => Some(carve_one(cidr, NODE_PREFIX_V6, &taken, pool)?), None => None, }; Ok((address4, address6)) } +/// Addresses in `cidr` that are not the pool's to hand out. +/// +/// The route server holds the whole block on-link, so the addresses that block +/// reserves are reserved here too: its network address, the route server's own +/// address immediately after it, and — on IPv4 — its broadcast address. +/// Handing any of them to a node would produce an address the route server +/// itself will not forward to. +pub fn reserved_addresses(cidr: &str) -> Vec { + let Ok(net) = cidr.parse::() else { + return vec![]; + }; + let mut out = vec![IpNetwork::from(net.network())]; + if let Some(server) = server_address(Some(cidr)) + && let Ok(addr) = bare_address(&server).parse::() + { + out.push(addr); + } + if let IpNetwork::V4(v4) = net { + out.push(IpNetwork::from(std::net::IpAddr::from(v4.broadcast()))); + } + out +} + fn carve_one(cidr: &str, prefix: u8, taken: &[IpNetwork], pool: &TunnelPool) -> Result { let block: IpNetwork = cidr.parse().map_err(|e| { anyhow!( @@ -291,13 +322,140 @@ fn carve_one(cidr: &str, prefix: u8, taken: &[IpNetwork], pool: &TunnelPool) -> pool.id ) })?; - let link = allocate_subnet(&block, prefix, taken).ok_or_else(|| { + let mut taken = taken.to_vec(); + taken.extend(reserved_addresses(cidr)); + let addr = allocate_subnet(&block, prefix, &taken).ok_or_else(|| { anyhow!( "Tunnel pool {} has no free /{prefix} left in {cidr}", pool.id ) })?; - Ok(peer_address(&link)) + Ok(addr.to_string()) +} + +/// What one pool's interface should have configured on its route server. +/// +/// Computed in one pass over the pool's tunnels because the three parts are +/// answers to the same question and must agree: an address without its peer is +/// a link to nowhere, a peer without its route drops the guest traffic it was +/// created to carry, and a route to a peer that is not there is a black hole. +#[derive(Debug, Clone, Default, PartialEq)] +pub struct PoolPlan { + /// Addresses on the interface — the route server's side of each link. + pub addresses: Vec, + /// One peer per realisable tunnel. + pub peers: Vec, + /// Guest prefixes routed down the interface. + pub routes: Vec, +} + +/// Work out what `pool`'s interface should look like. +/// +/// A tunnel that cannot be realised — disabled, or with no key presented yet — +/// contributes nothing at all, not an empty peer: half-configuring it would +/// give the node a link with no way to authenticate over it. +pub async fn plan_pool(db: &Arc, pool: &TunnelPool) -> Result { + let mut plan = PoolPlan::default(); + + // One address for the whole pool, carrying the block's prefix so every + // node in it is on-link. A per-node address would put one address on this + // interface for every node on the route server — thousands, on a /16 — to + // describe links that WireGuard, being layer 3 and point-to-point, does + // not need described. + plan.addresses.extend( + [ + server_address(pool.cidr4.as_deref()), + server_address(pool.cidr6.as_deref()), + ] + .into_iter() + .flatten(), + ); + + for tunnel in db.list_tunnels_in_pool(pool.id).await? { + if !tunnel.enabled { + continue; + } + let Some(key) = tunnel.peer_pubkey.as_deref() else { + continue; + }; + + // AllowedIPs is both the routing table for this peer and the + // anti-spoof boundary: WireGuard drops an inbound packet whose source + // is not listed, so a node cannot claim another node's guest address. + let mut allowed_ips: Vec = [ + host_address(tunnel.address4.as_deref()), + host_address(tunnel.address6.as_deref()), + ] + .into_iter() + .flatten() + .collect(); + + let guests = guest_addresses(db, &tunnel).await?; + allowed_ips.extend(guests.iter().cloned()); + plan.routes.extend(guests); + + plan.peers.push(WireguardPeer { + public_key: lnvps_api_common::wireguard_key_to_base64(key), + // Nodes dial out from behind NAT; the endpoint is learned from the + // handshake. Configuring a stale one would stop the peer from + // being reachable after the node's address changes. + endpoint: tunnel.peer_endpoint.clone(), + allowed_ips, + persistent_keepalive: tunnel.keepalive, + }); + } + Ok(plan) +} + +/// The public addresses assigned to the guests running on `tunnel`'s node. +/// +/// Empty for a tunnel that is not a marketplace node's, or a node with no +/// guests yet — a node is realised before it has customers, and the peer exists +/// so it can be given some. +async fn guest_addresses(db: &Arc, tunnel: &Tunnel) -> Result> { + let Some(node) = db.get_marketplace_node_by_tunnel(tunnel.id).await? else { + return Ok(vec![]); + }; + let Some(host) = db.get_marketplace_node_host(node.id).await? else { + return Ok(vec![]); + }; + + let mut out = Vec::new(); + for vm in db.list_vms_on_host(host.id).await? { + if vm.deleted { + continue; + } + for ip in db.list_vm_ip_assignments(vm.id).await? { + // A freed assignment must stop being routed here immediately: the + // address goes back in the pool and may already be somebody + // else's. + if ip.deleted { + continue; + } + if let Some(addr) = host_address(Some(&ip.ip)) { + out.push(addr); + } + } + } + out.sort(); + out.dedup(); + Ok(out) +} + +/// A single address as a host prefix (`/32` or `/128`). +/// +/// Accepts either a bare address or one already carrying a prefix, because +/// tunnel addresses are stored as CIDR and guest assignments as bare addresses. +fn host_address(addr: Option<&str>) -> Option { + let addr = addr?; + let ip: std::net::IpAddr = match addr.split_once('/') { + Some((a, _)) => a.parse().ok()?, + None => addr.parse().ok()?, + }; + Some(match ip { + std::net::IpAddr::V4(v4) => format!("{v4}/32"), + std::net::IpAddr::V6(v6) => format!("{v6}/128"), + }) } #[cfg(test)] @@ -404,20 +562,22 @@ mod tests { .unwrap() } - /// The first allocation takes the first link, both families, and gives the - /// node the second address of each — LNVPS keeps the first. + /// The first allocation takes one address of each family — the first the + /// block has left, after its own network address and the route server's. #[tokio::test] - async fn allocating_takes_the_first_free_link_in_both_families() { + async fn allocating_takes_the_first_free_address_in_both_families() { let (db, _mock, node, pool_id) = fixture().await; let allocation = allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); - assert_eq!(allocation.tunnel.address4.as_deref(), Some("10.66.0.1/31")); + assert_eq!(allocation.tunnel.address4.as_deref(), Some("10.66.0.2/32")); assert_eq!( allocation.tunnel.address6.as_deref(), - Some("fd00:66::1/127") + Some("fd00:66::2/128") ); - assert_eq!(allocation.gateway4().as_deref(), Some("10.66.0.0")); - assert_eq!(allocation.gateway6().as_deref(), Some("fd00:66::")); + // One gateway for the whole pool, not one per node: the route server + // holds a single address and every node in the block is on-link to it. + assert_eq!(allocation.gateway4().as_deref(), Some("10.66.0.1")); + assert_eq!(allocation.gateway6().as_deref(), Some("fd00:66::1")); assert_eq!(allocation.tunnel.pool_id, Some(pool_id)); assert_eq!(allocation.tunnel.kind, RouterTunnelKind::Wireguard); assert_eq!( @@ -437,13 +597,13 @@ mod tests { .await .unwrap() .unwrap(); - assert_eq!(host.ip, "10.66.0.1"); + assert_eq!(host.ip, "10.66.0.2"); assert!(!host.enabled); } - /// Two nodes must never share a link; the second takes the next one. + /// Two nodes must never share an address; the second takes the next one. #[tokio::test] - async fn a_second_node_gets_the_next_link() { + async fn a_second_node_gets_the_next_address() { let (db, _mock, first, _) = fixture().await; allocate_node_tunnel(&db, &first, &NODE_KEY).await.unwrap(); @@ -472,10 +632,10 @@ mod tests { let allocation = allocate_node_tunnel(&db, &second, &OTHER_KEY) .await .unwrap(); - assert_eq!(allocation.tunnel.address4.as_deref(), Some("10.66.0.3/31")); + assert_eq!(allocation.tunnel.address4.as_deref(), Some("10.66.0.3/32")); assert_eq!( allocation.tunnel.address6.as_deref(), - Some("fd00:66::3/127") + Some("fd00:66::3/128") ); } @@ -599,9 +759,10 @@ mod tests { #[tokio::test] async fn a_full_pool_falls_through_to_the_next_one() { let (db, mock, node, first_pool) = fixture().await; - // A /31 out of a /31 block: one link, and it is already taken. + // A /30 holds one placeable address once the network, route server and + // broadcast addresses are reserved — and it is already taken. let mut tiny = db.get_tunnel_pool(first_pool).await.unwrap(); - tiny.cidr4 = Some("10.66.0.0/31".to_string()); + tiny.cidr4 = Some("10.66.0.0/30".to_string()); tiny.cidr6 = None; db.update_tunnel_pool(&tiny).await.unwrap(); let user_id = db @@ -615,7 +776,7 @@ mod tests { router_id: Some(tiny.router_id), pool_id: Some(tiny.id), name: "squatter".to_string(), - address4: Some("10.66.0.1/31".to_string()), + address4: Some("10.66.0.2/32".to_string()), enabled: true, ..Default::default() }) @@ -625,7 +786,7 @@ mod tests { let spare = pool(&db, &mock, "10.77.0.0/24", None, "wg-mkt1").await; let allocation = allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); assert_eq!(allocation.tunnel.pool_id, Some(spare)); - assert_eq!(allocation.tunnel.address4.as_deref(), Some("10.77.0.1/31")); + assert_eq!(allocation.tunnel.address4.as_deref(), Some("10.77.0.2/32")); } /// A pool on a different router than the tunnel claims is a peer @@ -667,13 +828,13 @@ mod tests { let allocation = allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); assert_eq!(allocation.tunnel.address4, None); - assert_eq!(allocation.control_address().unwrap(), "[fd00:66::1]"); + assert_eq!(allocation.control_address().unwrap(), "[fd00:66::2]"); let host = db .get_marketplace_node_host(node.id) .await .unwrap() .unwrap(); - assert_eq!(host.ip, "[fd00:66::1]"); + assert_eq!(host.ip, "[fd00:66::2]"); } /// Reading back an allocation is how the node re-reads its configuration @@ -691,22 +852,39 @@ mod tests { assert_eq!(read.pool.mtu, 1420); } - /// The last link in a block is still a link. Deriving the peer's side by - /// setting the low bit is what makes that true without an overflow case. + /// The route server takes one address for the whole pool, and the block's + /// own reserved addresses are not the pool's to hand out: giving a node the + /// network or broadcast address of a block the route server holds on-link + /// produces an address it will not forward to. #[test] - fn the_top_of_the_address_space_still_yields_a_peer_address() { + fn the_block_keeps_its_reserved_addresses() { assert_eq!( - peer_address(&"255.255.255.254/31".parse().unwrap()), - "255.255.255.255/31" + server_address(Some("10.66.0.0/16")).as_deref(), + Some("10.66.0.1/16") ); assert_eq!( - peer_address( - &"ffff:ffff:ffff:ffff:ffff:ffff:ffff:fffe/127" - .parse() - .unwrap() - ), - "ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff/127" + server_address(Some("fd00:66::/48")).as_deref(), + Some("fd00:66::1/48") ); + + let reserved: Vec = reserved_addresses("10.66.0.0/16") + .iter() + .map(|a| a.ip().to_string()) + .collect(); + assert_eq!(reserved, ["10.66.0.0", "10.66.0.1", "10.66.255.255"]); + + // IPv6 has no broadcast address, but the subnet-router anycast address + // is still not a node's. + let reserved: Vec = reserved_addresses("fd00:66::/64") + .iter() + .map(|a| a.ip().to_string()) + .collect(); + assert_eq!(reserved, ["fd00:66::", "fd00:66::1"]); + + // Nothing to reserve out of a block nobody can parse; the allocator + // reports that against the pool. + assert!(reserved_addresses("not-a-cidr").is_empty()); + assert_eq!(server_address(None), None); } /// A tunnel with no address is unreachable, and saying so beats handing @@ -739,4 +917,181 @@ mod tests { .expect_err("an unparseable block allocated an address"); assert!(format!("{err}").contains("unparseable block"), "{err}"); } + + /// A realised peer is the anti-spoof boundary: `AllowedIPs` is the node's + /// own inner addresses plus exactly the guest addresses LNVPS assigned to + /// it, so a node cannot source traffic as another node's customer. + #[tokio::test] + async fn a_peer_allows_the_node_its_own_addresses_and_its_guests() { + let (db, mock, node, pool_id) = fixture().await; + let allocation = allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let host = db + .get_marketplace_node_host(node.id) + .await + .unwrap() + .unwrap(); + add_guest(&db, &mock, host.id, &["203.0.113.5", "2001:db8::5"]).await; + + let pool = db.get_tunnel_pool(pool_id).await.unwrap(); + let plan = plan_pool(&db, &pool).await.unwrap(); + + assert_eq!(plan.peers.len(), 1); + let peer = &plan.peers[0]; + assert_eq!( + peer.public_key, + lnvps_api_common::wireguard_key_to_base64(&NODE_KEY) + ); + assert_eq!( + peer.allowed_ips, + vec![ + "10.66.0.2/32".to_string(), + "fd00:66::2/128".to_string(), + "2001:db8::5/128".to_string(), + "203.0.113.5/32".to_string(), + ] + ); + assert_eq!(peer.persistent_keepalive, Some(25)); + + // One address for the whole pool, carrying the block's prefix so every + // node in it is on-link. A per-node address would put one address on + // this interface for every node on the route server. + assert_eq!( + plan.addresses, + vec!["10.66.0.1/24".to_string(), "fd00:66::1/64".to_string()] + ); + // ...and a route for each guest address, because AllowedIPs picks the + // peer for a packet already headed down the tunnel, it does not put it + // there. + assert_eq!( + plan.routes, + vec!["2001:db8::5/128".to_string(), "203.0.113.5/32".to_string()] + ); + assert_eq!(allocation.tunnel.pool_id, Some(pool_id)); + } + + /// A freed address must stop being routed to the node at once: it goes + /// straight back in the pool and may already be somebody else's. + #[tokio::test] + async fn a_released_guest_address_is_not_routed() { + let (db, mock, node, pool_id) = fixture().await; + allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let host = db + .get_marketplace_node_host(node.id) + .await + .unwrap() + .unwrap(); + let vm_id = add_guest(&db, &mock, host.id, &["203.0.113.5"]).await; + + // Release the address, leaving the VM in place. + { + let mut ips = mock.ip_assignments.lock().await; + for ip in ips.values_mut().filter(|i| i.vm_id == vm_id) { + ip.deleted = true; + } + } + let pool = db.get_tunnel_pool(pool_id).await.unwrap(); + let plan = plan_pool(&db, &pool).await.unwrap(); + assert_eq!(plan.routes, Vec::::new()); + assert_eq!(plan.peers[0].allowed_ips.len(), 2, "only the node's own"); + + // A deleted VM takes its addressing with it for the same reason. + { + let mut ips = mock.ip_assignments.lock().await; + for ip in ips.values_mut().filter(|i| i.vm_id == vm_id) { + ip.deleted = false; + } + let mut vms = mock.vms.lock().await; + if let Some(vm) = vms.get_mut(&vm_id) { + vm.deleted = true; + } + } + let plan = plan_pool(&db, &pool).await.unwrap(); + assert_eq!(plan.routes, Vec::::new()); + } + + /// A tunnel that cannot be realised contributes nothing at all, rather than + /// an empty peer: a link with no key is one the node cannot authenticate + /// over, and an address on it would be a link to nowhere. + #[tokio::test] + async fn an_unrealisable_tunnel_is_left_out_entirely() { + let (db, _mock, node, pool_id) = fixture().await; + let allocation = allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let pool = db.get_tunnel_pool(pool_id).await.unwrap(); + + for broken in [ + Tunnel { + enabled: false, + ..allocation.tunnel.clone() + }, + Tunnel { + peer_pubkey: None, + ..allocation.tunnel.clone() + }, + ] { + db.update_tunnel(&broken).await.unwrap(); + let plan = plan_pool(&db, &pool).await.unwrap(); + assert!(plan.peers.is_empty()); + assert!(plan.routes.is_empty()); + // The interface still holds the pool's address: it exists whether + // or not anything has been placed in it yet. + assert_eq!(plan.addresses.len(), 2); + } + } + + /// A tunnel that is not a node's — a customer VPN carved from the same + /// pool later — is still a peer, just one with no guests behind it. + #[tokio::test] + async fn a_tunnel_with_no_node_behind_it_is_still_a_peer() { + let (db, mock, _node, pool_id) = fixture().await; + let user_id = db.upsert_user(&[9u8; 32]).await.unwrap(); + db.insert_tunnel(&Tunnel { + kind: RouterTunnelKind::Wireguard, + user_id, + router_id: Some(db.get_tunnel_pool(pool_id).await.unwrap().router_id), + pool_id: Some(pool_id), + name: "vpn".to_string(), + peer_pubkey: Some(OTHER_KEY.to_vec()), + address4: Some("10.66.9.1/32".to_string()), + enabled: true, + ..Default::default() + }) + .await + .unwrap(); + + let pool = db.get_tunnel_pool(pool_id).await.unwrap(); + let plan = plan_pool(&db, &pool).await.unwrap(); + assert_eq!(plan.peers.len(), 1); + assert_eq!(plan.peers[0].allowed_ips, vec!["10.66.9.1/32".to_string()]); + assert!(plan.routes.is_empty()); + let _ = mock; + } + + /// Give `host` a VM holding `ips`. Written through the mock's maps because + /// a VM needs a template, an image and a disk that this test does not care + /// about. + async fn add_guest(db: &Arc, mock: &MockDb, host_id: u64, ips: &[&str]) -> u64 { + let vm_id = { + let mut vms = mock.vms.lock().await; + let id = vms.keys().max().copied().unwrap_or(0) + 1; + vms.insert( + id, + lnvps_db::Vm { + id, + host_id, + ..Default::default() + }, + ); + id + }; + for ip in ips { + db.insert_vm_ip_assignment(&lnvps_db::VmIpAssignment { + vm_id, + ip: ip.to_string(), + ..Default::default() + }) + .await + .unwrap(); + } + vm_id + } } diff --git a/lnvps_api/src/router/linux_ssh.rs b/lnvps_api/src/router/linux_ssh.rs index 5c538b1b..00c8ac91 100644 --- a/lnvps_api/src/router/linux_ssh.rs +++ b/lnvps_api/src/router/linux_ssh.rs @@ -28,8 +28,19 @@ pub struct LinuxSshRouter { interface: String, /// SSH private key (PEM) key: String, + /// Stands in for the SSH connection under test. + /// + /// These methods run commands as root on somebody else's route server, so + /// what is worth asserting is the exact command issued — which needs the + /// transport replaced, not mocked around. + #[cfg(test)] + exec: Option, } +/// A stand-in for running a command over SSH. +#[cfg(test)] +type ExecFn = std::sync::Arc OpResult + Send + Sync>; + impl LinuxSshRouter { /// Build a router from the stored config `url` and `token`. pub fn new(url: &str, key: &str) -> Result { @@ -57,9 +68,23 @@ impl LinuxSshRouter { username, interface, key: key.to_string(), + #[cfg(test)] + exec: None, }) } + /// A router whose commands are handled by `exec` instead of an SSH session. + #[cfg(test)] + fn with_exec(exec: ExecFn) -> Self { + Self { + host: "test:22".to_string(), + username: "root".to_string(), + interface: "eth0".to_string(), + key: String::new(), + exec: Some(exec), + } + } + /// Open a fresh SSH connection for a single operation. /// /// Connecting per-operation keeps the router `Send + Sync` without holding a @@ -76,6 +101,10 @@ impl LinuxSshRouter { /// Run a command, mapping connection failures to transient errors and /// non-zero exits to fatal errors. Returns stdout on success. async fn exec_checked(&self, cmd: &str) -> OpResult { + #[cfg(test)] + if let Some(exec) = &self.exec { + return exec(cmd); + } let mut client = match self.connect().await { Ok(c) => c, Err(e) => op_transient!(e), @@ -618,6 +647,90 @@ impl TunnelRouter for LinuxSshRouter { Ok(()) } + async fn set_tunnel_peer(&self, interface: &str, peer: &WireguardPeer) -> OpResult<()> { + // `wg set ... peer` is additive and idempotent: it creates the peer or + // rewrites the fields given, and leaves the rest of the interface + // alone. That is the whole reason peers are pushed one at a time rather + // than through `update_tunnel`, which recreates the interface. + self.exec_checked(&format!("sh -c {}", shq(&wg_peer_script(interface, peer)))) + .await?; + Ok(()) + } + + async fn remove_tunnel_peer(&self, interface: &str, public_key: &str) -> OpResult<()> { + // Removing a peer that is not there succeeds, which is what makes this + // safe to retry after a partial teardown. + self.exec_checked(&format!( + "wg set {} peer {} remove", + shq(interface), + shq(public_key) + )) + .await?; + Ok(()) + } + + async fn sync_tunnel_addresses(&self, interface: &str, addresses: &[String]) -> OpResult<()> { + let out = self + .exec_checked(&format!("ip -j addr show dev {}", shq(interface))) + .await?; + let observed = match parse_addr_show(&out) { + Ok(a) => a, + Err(e) => op_fatal!("Failed to parse ip addr output: {}", e), + }; + + let script = sync_set_script( + &observed, + addresses, + |a| format!("ip addr add {} dev {}", shq(a), shq(interface)), + // A link-local IPv6 address is put there by the kernel, not by us. + // Deleting it would be deleting the interface's ability to talk to + // itself, on every single sync. + |a| format!("ip addr del {} dev {}", shq(a), shq(interface)), + |a| !a.starts_with("fe80:"), + ); + if let Some(script) = script { + self.exec_checked(&format!("sh -c {}", shq(&script))) + .await?; + } + Ok(()) + } + + async fn sync_tunnel_routes(&self, interface: &str, prefixes: &[String]) -> OpResult<()> { + // Both families have to be asked for separately: `ip route show` is + // IPv4 only, and a v6 guest prefix would otherwise look like a route + // that is missing on every sync and never stops being re-added. + let mut observed = Vec::new(); + for family in ["-4", "-6"] { + let out = self + .exec_checked(&format!( + "ip {} -j route show dev {}", + family, + shq(interface) + )) + .await?; + match parse_route_show(&out) { + Ok(r) => observed.extend(r), + Err(e) => op_fatal!("Failed to parse ip route output: {}", e), + } + } + + let script = sync_set_script( + &observed, + prefixes, + |p| format!("ip route replace {} dev {}", shq(p), shq(interface)), + |p| format!("ip route del {} dev {}", shq(p), shq(interface)), + // The route the kernel installs for the interface's own address is + // the link itself. Removing it would break the link to keep a list + // tidy. + |p| !p.contains("/31") && !p.contains("/127"), + ); + if let Some(script) = script { + self.exec_checked(&format!("sh -c {}", shq(&script))) + .await?; + } + Ok(()) + } + async fn tunnel_traffic(&self) -> OpResult> { let out = self.exec_checked("ip -s -d -j link show").await?; let links: Vec = match serde_json::from_str(&out) { @@ -643,6 +756,94 @@ impl TunnelRouter for LinuxSshRouter { } } +/// The commands that turn `observed` into `desired`, or `None` when they +/// already agree. +/// +/// Returning `None` rather than a no-op script keeps an unchanged interface +/// from being touched at all, which matters because these run on somebody +/// else's route server on every reconcile. +/// +/// `is_ours` marks the entries this code is allowed to delete: an interface +/// carries state the kernel put there, and a sync that removes everything it +/// did not add would fight the kernel forever. +fn sync_set_script( + observed: &[String], + desired: &[String], + add: impl Fn(&str) -> String, + del: impl Fn(&str) -> String, + is_ours: impl Fn(&str) -> bool, +) -> Option { + let mut parts = Vec::new(); + for want in desired { + if !observed.iter().any(|o| o == want) { + parts.push(add(want)); + } + } + for have in observed { + if is_ours(have) && !desired.iter().any(|d| d == have) { + parts.push(del(have)); + } + } + if parts.is_empty() { + None + } else { + Some(parts.join(" && ")) + } +} + +/// Build the `wg set` command for a single peer. +/// +/// `allowed-ips` *replaces* the peer's list rather than adding to it, so this +/// narrows a peer's reach as well as widening it — which is what makes it safe +/// to use as the anti-spoof boundary. +fn wg_peer_script(interface: &str, p: &WireguardPeer) -> String { + let mut script = String::new(); + script.push_str(&format!( + "wg set {} peer {}", + shq(interface), + shq(&p.public_key) + )); + if let Some(e) = &p.endpoint { + script.push_str(&format!(" endpoint {}", shq(e))); + } + if !p.allowed_ips.is_empty() { + script.push_str(&format!(" allowed-ips {}", shq(&p.allowed_ips.join(",")))); + } + if let Some(k) = p.persistent_keepalive { + script.push_str(&format!(" persistent-keepalive {}", k)); + } + script +} + +/// Addresses from `ip -j addr show dev X`, as CIDR. +fn parse_addr_show(json: &str) -> Result, serde_json::Error> { + let links: Vec = serde_json::from_str(json)?; + Ok(links + .into_iter() + .flat_map(|l| l.addr_info) + .map(|a| format!("{}/{}", a.local, a.prefixlen)) + .collect()) +} + +/// Destination prefixes from `ip -j route show dev X`. +fn parse_route_show(json: &str) -> Result, serde_json::Error> { + let routes: Vec = serde_json::from_str(json)?; + Ok(routes + .into_iter() + .map(|r| { + // iproute2 renders a host route as a bare address. Normalising it + // here means a desired "/32" is not re-added on every sync. + if r.dst.contains('/') || r.dst == "default" { + r.dst + } else if r.dst.contains(':') { + format!("{}/128", r.dst) + } else { + format!("{}/32", r.dst) + } + }) + .collect()) +} + /// Build a `wg set` command chain for a WireGuard interface configuration. fn wg_set_script(name: &str, c: &WireguardConfig) -> String { let mut parts = Vec::new(); @@ -728,6 +929,25 @@ fn none_if_marker(s: &str) -> Option<&str> { } } +/// One entry from `ip -j addr show` +#[derive(Debug, Clone, Deserialize)] +struct IpAddrLink { + #[serde(default)] + addr_info: Vec, +} + +#[derive(Debug, Clone, Deserialize)] +struct IpAddrInfo { + local: String, + prefixlen: u8, +} + +/// One entry from `ip -j route show` +#[derive(Debug, Clone, Deserialize)] +struct IpRouteEntry { + dst: String, +} + /// One entry from `ip -s -d -j link show` #[derive(Debug, Clone, Deserialize)] struct IpLink { @@ -793,6 +1013,7 @@ impl IpNeighEntry { #[cfg(test)] mod tests { use super::*; + use std::sync::Mutex; #[test] fn test_new_parses_url() -> Result<()> { @@ -1066,4 +1287,242 @@ mod tests { assert_eq!(arp[0].mac_address, "aa:bb:cc:dd:ee:ff"); assert_eq!(arp[0].id.as_deref(), Some("10.0.0.5")); } + + /// A peer push must not disturb the rest of the interface: it is one `wg + /// set peer`, not a re-apply, because re-applying recreates the interface + /// and drops every other node on it. + + /// A router that records what it was asked to run and answers `ip` queries + /// from `state`. + fn recorded(state: &'static str) -> (LinuxSshRouter, std::sync::Arc>>) { + let log = std::sync::Arc::new(Mutex::new(Vec::new())); + let sink = log.clone(); + let r = LinuxSshRouter::with_exec(std::sync::Arc::new(move |cmd: &str| { + sink.lock().unwrap().push(cmd.to_string()); + // Only the IPv4 query answers with state: a fixture that returned + // the same rows for both families would make a v4 route look like + // a v6 one as well. + Ok(if cmd.contains("-6 -j route show") { + "[]".to_string() + } else if cmd.contains("show") { + state.to_string() + } else { + String::new() + }) + })); + (r, log) + } + + /// A peer push is one `wg set peer` and nothing else: it must not recreate + /// the interface, which on this backend would drop every other node on it. + #[tokio::test] + async fn test_set_tunnel_peer_only_touches_that_peer() { + let (r, log) = recorded("[]"); + r.set_tunnel_peer( + "wgln1", + &WireguardPeer { + public_key: "cGVlcg==".to_string(), + allowed_ips: vec!["10.66.0.1/32".to_string()], + persistent_keepalive: Some(25), + endpoint: None, + }, + ) + .await + .unwrap(); + + let ran = log.lock().unwrap().clone(); + assert_eq!(ran.len(), 1); + assert!(ran[0].contains("wg set"), "{}", ran[0]); + assert!(ran[0].contains("peer"), "{}", ran[0]); + assert!(ran[0].contains("cGVlcg=="), "{}", ran[0]); + assert!(ran[0].contains("allowed-ips"), "{}", ran[0]); + assert!(ran[0].contains("persistent-keepalive 25"), "{}", ran[0]); + assert!(!ran[0].contains("ip link"), "{}", ran[0]); + } + + /// Removing a peer that is not there succeeds, which is what makes the + /// teardown safe to retry after a partial failure. + #[tokio::test] + async fn test_remove_tunnel_peer_is_one_command() { + let (r, log) = recorded("[]"); + r.remove_tunnel_peer("wgln1", "cGVlcg==").await.unwrap(); + assert_eq!( + log.lock().unwrap().clone(), + vec!["wg set 'wgln1' peer 'cGVlcg==' remove".to_string()] + ); + } + + /// Only the difference is applied, and the kernel's own link-local address + /// is left alone — deleting it would break the interface, on every sync. + #[tokio::test] + async fn test_sync_tunnel_addresses_applies_only_the_difference() { + let state = r#"[{"ifname":"wgln1","addr_info":[ + {"local":"10.66.0.0","prefixlen":31}, + {"local":"10.66.0.2","prefixlen":31}, + {"local":"fe80::1","prefixlen":64} + ]}]"#; + let (r, log) = recorded(state); + r.sync_tunnel_addresses( + "wgln1", + &["10.66.0.0/31".to_string(), "10.66.0.4/31".to_string()], + ) + .await + .unwrap(); + + let ran = log.lock().unwrap().clone(); + assert_eq!(ran.len(), 2, "{ran:?}"); + assert!(ran[1].contains("ip addr add"), "{}", ran[1]); + assert!(ran[1].contains("10.66.0.4/31"), "{}", ran[1]); + assert!(ran[1].contains("ip addr del"), "{}", ran[1]); + assert!(ran[1].contains("10.66.0.2/31"), "{}", ran[1]); + assert!( + !ran[1].contains("fe80"), + "the kernel's own address was deleted" + ); + } + + /// An interface that already matches is not touched at all: this runs on + /// every reconcile, on a machine LNVPS does not own. + #[tokio::test] + async fn test_sync_tunnel_addresses_is_silent_when_correct() { + let state = r#"[{"ifname":"wgln1","addr_info":[{"local":"10.66.0.0","prefixlen":31}]}]"#; + let (r, log) = recorded(state); + r.sync_tunnel_addresses("wgln1", &["10.66.0.0/31".to_string()]) + .await + .unwrap(); + assert_eq!(log.lock().unwrap().len(), 1, "only the query"); + } + + /// Both families have to be asked for separately, or a v6 guest prefix + /// looks absent on every sync and is re-added forever. + #[tokio::test] + async fn test_sync_tunnel_routes_covers_both_families() { + let (r, log) = recorded("[]"); + r.sync_tunnel_routes("wgln1", &["203.0.113.5/32".to_string()]) + .await + .unwrap(); + + let ran = log.lock().unwrap().clone(); + assert!(ran[0].contains("ip -4 -j route show"), "{}", ran[0]); + assert!(ran[1].contains("ip -6 -j route show"), "{}", ran[1]); + assert!(ran[2].contains("ip route replace"), "{}", ran[2]); + assert!(ran[2].contains("203.0.113.5/32"), "{}", ran[2]); + assert!(ran[2].contains("wgln1"), "{}", ran[2]); + } + + /// The link route the kernel installs for the interface's own /31 is the + /// link itself. Removing it to tidy a list would break the tunnel. + #[tokio::test] + async fn test_sync_tunnel_routes_keeps_the_link_route() { + let (r, log) = recorded(r#"[{"dst":"10.66.0.0/31"},{"dst":"198.51.100.9"}]"#); + r.sync_tunnel_routes("wgln1", &[]).await.unwrap(); + + let ran = log.lock().unwrap().clone(); + let applied = ran.last().unwrap(); + assert!(applied.contains("ip route del"), "{applied}"); + assert!(applied.contains("198.51.100.9/32"), "{applied}"); + assert!(!applied.contains("10.66.0.0/31"), "{applied}"); + } + + #[test] + fn test_wg_peer_script_touches_one_peer() { + let script = wg_peer_script( + "wgln1", + &WireguardPeer { + public_key: "cGVlcg==".to_string(), + endpoint: None, + allowed_ips: vec!["10.66.0.1/32".to_string(), "203.0.113.5/32".to_string()], + persistent_keepalive: Some(25), + }, + ); + assert_eq!( + script, + "wg set 'wgln1' peer 'cGVlcg==' allowed-ips '10.66.0.1/32,203.0.113.5/32' \ + persistent-keepalive 25" + ); + // Nothing that recreates or reconfigures the interface itself. + assert!(!script.contains("ip link")); + assert!(!script.contains("private-key")); + } + + /// A peer with nothing but a key is still a valid statement: the node has + /// asked for a tunnel but has no addresses routed to it yet. + #[test] + fn test_wg_peer_script_without_optional_fields() { + let script = wg_peer_script( + "wgln1", + &WireguardPeer { + public_key: "cGVlcg==".to_string(), + ..Default::default() + }, + ); + assert_eq!(script, "wg set 'wgln1' peer 'cGVlcg=='"); + } + + /// An interface that already matches must not be touched at all: these + /// commands run on somebody else's route server on every reconcile. + #[test] + fn test_sync_set_script_is_silent_when_nothing_changed() { + let observed = vec!["10.66.0.0/31".to_string()]; + let desired = vec!["10.66.0.0/31".to_string()]; + assert_eq!( + sync_set_script( + &observed, + &desired, + |a| format!("add {a}"), + |a| format!("del {a}"), + |_| true + ), + None + ); + } + + /// Both directions in one pass, and only over entries this code owns. + #[test] + fn test_sync_set_script_adds_and_removes() { + let observed = vec![ + "10.66.0.0/31".to_string(), + "10.66.0.2/31".to_string(), + "fe80::1/64".to_string(), + ]; + let desired = vec!["10.66.0.0/31".to_string(), "10.66.0.4/31".to_string()]; + let script = sync_set_script( + &observed, + &desired, + |a| format!("add {a}"), + |a| format!("del {a}"), + // The kernel's own link-local address is not ours to delete. + |a| !a.starts_with("fe80:"), + ) + .unwrap(); + assert_eq!(script, "add 10.66.0.4/31 && del 10.66.0.2/31"); + } + + #[test] + fn test_parse_addr_show() { + let json = r#"[{"ifname":"wgln1","addr_info":[ + {"family":"inet","local":"10.66.0.0","prefixlen":31}, + {"family":"inet6","local":"fd00:66::","prefixlen":127} + ]}]"#; + assert_eq!( + parse_addr_show(json).unwrap(), + vec!["10.66.0.0/31".to_string(), "fd00:66::/127".to_string()] + ); + } + + /// iproute2 renders a host route as a bare address. Normalising it is what + /// stops a desired `/32` from looking absent on every single sync and being + /// re-added forever. + #[test] + fn test_parse_route_show_normalises_host_routes() { + let json = r#"[{"dst":"203.0.113.5"},{"dst":"2001:db8::5"},{"dst":"198.51.100.0/24"}]"#; + assert_eq!( + parse_route_show(json).unwrap(), + vec![ + "203.0.113.5/32".to_string(), + "2001:db8::5/128".to_string(), + "198.51.100.0/24".to_string() + ] + ); + } } diff --git a/lnvps_api/src/router/mikrotik.rs b/lnvps_api/src/router/mikrotik.rs index 678c31db..4cdbe322 100644 --- a/lnvps_api/src/router/mikrotik.rs +++ b/lnvps_api/src/router/mikrotik.rs @@ -746,4 +746,21 @@ mod tunnel_tests { assert_eq!(p2.keepalive_secs(), Some(30)); assert_eq!(p2.endpoint(), None); } + + /// A marketplace route server is a Linux box: peers, interface addresses + /// and per-guest routes are pushed with `wg` and `ip`. A Mikrotik carrying + /// a tunnel pool would silently accept the pool and configure nothing, so + /// the unimplemented half of the capability says so instead. + #[tokio::test] + async fn peer_level_operations_are_refused_rather_than_ignored() { + let r = MikrotikRouter::new("http://10.0.0.1", "admin", "pw"); + assert!( + r.set_tunnel_peer("wgln1", &WireguardPeer::default()) + .await + .is_err() + ); + assert!(r.remove_tunnel_peer("wgln1", "key").await.is_err()); + assert!(r.sync_tunnel_addresses("wgln1", &[]).await.is_err()); + assert!(r.sync_tunnel_routes("wgln1", &[]).await.is_err()); + } } diff --git a/lnvps_api/src/router/mod.rs b/lnvps_api/src/router/mod.rs index 146f642c..f30694e1 100644 --- a/lnvps_api/src/router/mod.rs +++ b/lnvps_api/src/router/mod.rs @@ -1,5 +1,6 @@ use anyhow::{Context, Result, ensure}; use async_trait::async_trait; +use lnvps_api_common::op_fatal; use lnvps_api_common::retry::OpResult; use lnvps_db::{LNVpsDb, RouterKind, Vm, VmIpAssignment}; use std::sync::Arc; @@ -161,6 +162,56 @@ pub trait TunnelRouter: Send + Sync { async fn set_tunnel_enabled(&self, id: &str, enabled: bool) -> OpResult<()>; /// Report per-tunnel rx/tx byte counters async fn tunnel_traffic(&self) -> OpResult>; + + /// Add or update a single WireGuard peer on `interface`. + /// + /// Separate from [`update_tunnel`](Self::update_tunnel) because that + /// re-applies the whole interface — on the Linux backend by recreating it, + /// which drops every other peer with it. One node getting a new guest + /// address must not cut every other node on the same route server. + /// + /// Idempotent: a peer that is already configured this way is left as it is. + async fn set_tunnel_peer(&self, interface: &str, peer: &WireguardPeer) -> OpResult<()> { + let _ = (interface, peer); + op_fatal!("This router backend cannot manage individual tunnel peers") + } + + /// Remove a WireGuard peer from `interface` by its public key. + /// + /// Idempotent: a peer that is not there is the desired state. + async fn remove_tunnel_peer(&self, interface: &str, public_key: &str) -> OpResult<()> { + let _ = (interface, public_key); + op_fatal!("This router backend cannot manage individual tunnel peers") + } + + /// Make the addresses on `interface` exactly `addresses` (CIDR). + /// + /// A WireGuard interface with no address is a tunnel that terminates + /// nowhere: the peer's default route points at *some* address on this side, + /// and it has to exist for anything to answer. + /// + /// Declarative rather than add/remove, because the desired set is what the + /// allocator knows; working out which individual addresses drifted is this + /// method's job, not its caller's. + async fn sync_tunnel_addresses(&self, interface: &str, addresses: &[String]) -> OpResult<()> { + let _ = (interface, addresses); + op_fatal!("This router backend cannot manage tunnel interface addresses") + } + + /// Make the routes pointing down `interface` exactly `prefixes` (CIDR). + /// + /// WireGuard's `AllowedIPs` decides which *peer* a packet already destined + /// for the tunnel belongs to; it does not put the packet on the tunnel in + /// the first place. Without a route, a guest's return traffic arrives at the + /// route server and is dropped as unroutable. + /// + /// No next hop: the interface plus `AllowedIPs` is enough to select the + /// peer, and naming a gateway would be a second place for the peer's + /// address to be recorded. + async fn sync_tunnel_routes(&self, interface: &str, prefixes: &[String]) -> OpResult<()> { + let _ = (interface, prefixes); + op_fatal!("This router backend cannot manage tunnel routes") + } } /// The kind of a tunnel interface diff --git a/lnvps_api/src/worker.rs b/lnvps_api/src/worker.rs index 9c0fb80c..669728ac 100644 --- a/lnvps_api/src/worker.rs +++ b/lnvps_api/src/worker.rs @@ -136,6 +136,53 @@ struct HostInfoOutput { gpu_features: Vec, } +/// What a tunnel pool's route server disagreed with the database about. +/// +/// Kept as three lists rather than a count because they mean different things: +/// a peer that is *missing* was configured and is gone, a *changed* one is +/// carrying the wrong anti-spoof list, and an *unclaimed* one is a key on an +/// LNVPS interface that no allocation accounts for. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct TunnelPeerDrift { + /// Allocated peers the route server did not have + pub missing: Vec, + /// Peers whose allowed IPs no longer matched their allocation + pub changed: Vec, + /// Peers on the interface that no tunnel claims + pub unclaimed: Vec, +} + +impl TunnelPeerDrift { + pub fn is_empty(&self) -> bool { + self.missing.is_empty() && self.changed.is_empty() && self.unclaimed.is_empty() + } +} + +impl std::fmt::Display for TunnelPeerDrift { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + f, + "{} missing, {} changed, {} unclaimed", + self.missing.len(), + self.changed.len(), + self.unclaimed.len() + ) + } +} + +/// Whether two peers permit the same set of addresses. +/// +/// Compared as a set: `wg` reports allowed IPs in its own order, and treating +/// that as a difference would rewrite a working peer's anti-spoof list on every +/// single reconcile. +fn same_allowed_ips(a: &crate::router::WireguardPeer, b: &crate::router::WireguardPeer) -> bool { + let mut x: Vec<&String> = a.allowed_ips.iter().collect(); + let mut y: Vec<&String> = b.allowed_ips.iter().collect(); + x.sort(); + y.sort(); + x == y +} + /// Primary background worker logic /// Handles deleting expired VMs and sending notifications #[derive(Clone)] @@ -728,6 +775,26 @@ impl Worker { router_id, e ), } + + // Reconcile every pool this router terminates. This is where drift + // is actually caught: a peer wiped by a reboot, a stale key left + // behind, or a guest address assigned since the last push. Doing it + // on the existing router poll rather than on every VM change keeps + // guest addressing correct without wiring a route-server call into + // the provisioning path. + match self.db.list_tunnel_pools(None).await { + Ok(pools) => { + for pool in pools + .iter() + .filter(|p| p.router_id == router_id && p.enabled) + { + if let Err(e) = self.reconcile_tunnel_peers(pool.id).await { + warn!("Failed to reconcile tunnel pool {}: {}", pool.id, e); + } + } + } + Err(e) => warn!("Failed to list tunnel pools for router {router_id}: {e}"), + } } // BGP: refresh cached session state (no traffic counters) @@ -972,6 +1039,158 @@ impl Worker { } } } + + // Whatever happened above, the interface now has to carry the peers + // that were allocated from this pool. This matters most in the case the + // push above just created or re-applied it: on Linux that is a fresh + // interface with no peers at all, and every node on it is cut until + // they are put back. + self.reconcile_tunnel_peers(pool.id).await?; + Ok(()) + } + + /// Reconcile the peers, addresses and routes on a pool's interface against + /// the tunnels allocated from it. + /// + /// The `tunnel` table is the desired state and the router is the observed + /// one, exactly as with host state. A peer that has vanished from a route + /// server is drift to put back and report, not an allocation to forget: + /// forgetting it would hand the node's addresses to somebody else while the + /// node still believes they are its own. + /// + /// Returns what had drifted, so a caller running this on a schedule can say + /// whether anything was wrong rather than only that it ran. + pub async fn reconcile_tunnel_peers(&self, pool_id: u64) -> Result { + let pool = self.db.get_tunnel_pool(pool_id).await?; + let router = crate::router::get_router(&self.db, pool.router_id) + .await + .map_err(|e| anyhow!("failed to load router {}: {}", pool.router_id, e))?; + let tr = router.tunnel().context("router does not support tunnels")?; + let interface = pool.interface(); + + let observed = tr + .list_tunnels() + .await + .map_err(|e| anyhow!("failed to list tunnels: {}", e))? + .into_iter() + .find(|t| t.name == interface); + // Peers are configured *on* an interface, so there is nothing to + // reconcile against until it exists. Creating it here would duplicate + // `sync_tunnel_pool` and hide the fact that it never ran. + let Some(observed) = observed else { + bail!( + "Tunnel pool {pool_id}'s interface {interface} is not configured on router {}; \ + run SyncTunnelPool first", + pool.router_id + ); + }; + let observed_peers = match &observed.config { + crate::router::TunnelConfig::Wireguard(c) => c.peers.clone(), + _ => bail!("Tunnel pool {pool_id}'s interface {interface} is not a WireGuard tunnel"), + }; + + let plan = crate::provisioner::plan_pool(&self.db, &pool).await?; + let mut drift = TunnelPeerDrift::default(); + + for want in &plan.peers { + match observed_peers + .iter() + .find(|p| p.public_key == want.public_key) + { + // Allowed IPs are compared as a set: `wg` reports them in its + // own order, and re-pushing on every reconcile because of that + // would rewrite the anti-spoof list of a working peer forever. + Some(have) if same_allowed_ips(have, want) => continue, + Some(_) => drift.changed.push(want.public_key.clone()), + None => drift.missing.push(want.public_key.clone()), + } + tr.set_tunnel_peer(&interface, want) + .await + .map_err(|e| anyhow!("failed to configure peer on {interface}: {}", e))?; + } + + for have in &observed_peers { + if plan.peers.iter().any(|p| p.public_key == have.public_key) { + continue; + } + // LNVPS owns `wgln*` interfaces outright, so a peer no tunnel + // claims is either a revoked allocation that was never cleaned up + // or somebody else's key on our route server. Both are removed. + drift.unclaimed.push(have.public_key.clone()); + tr.remove_tunnel_peer(&interface, &have.public_key) + .await + .map_err(|e| anyhow!("failed to remove peer from {interface}: {}", e))?; + } + + tr.sync_tunnel_addresses(&interface, &plan.addresses) + .await + .map_err(|e| anyhow!("failed to configure addresses on {interface}: {}", e))?; + tr.sync_tunnel_routes(&interface, &plan.routes) + .await + .map_err(|e| anyhow!("failed to configure routes on {interface}: {}", e))?; + + if !drift.is_empty() { + warn!( + "Tunnel pool {pool_id} on router {} had drifted: {drift}", + pool.router_id + ); + } + Ok(drift) + } + + /// Push one node's peer onto its route server. + /// + /// Used when a single allocation changes — a node asking for its tunnel, a + /// guest getting an address — so it does not wait behind a reconcile of + /// every other node on the same route server. + pub async fn sync_node_tunnel(&self, tunnel_id: u64) -> Result<()> { + let tunnel = self.db.get_tunnel(tunnel_id).await?; + let pool_id = tunnel.pool_id.ok_or_else(|| { + anyhow!("Tunnel {tunnel_id} was not allocated from a pool, so there is no interface") + })?; + let pool = self.db.get_tunnel_pool(pool_id).await?; + let router = crate::router::get_router(&self.db, pool.router_id) + .await + .map_err(|e| anyhow!("failed to load router {}: {}", pool.router_id, e))?; + let tr = router.tunnel().context("router does not support tunnels")?; + let interface = pool.interface(); + + // The peer's own share of the pool plan, rather than a second + // calculation of what one tunnel needs: the addresses and routes are + // per-interface, so one node's change is applied by re-stating the + // whole interface's addressing, and only its own peer is pushed. + let plan = crate::provisioner::plan_pool(&self.db, &pool).await?; + let key = tunnel + .peer_pubkey + .as_deref() + .map(lnvps_api_common::wireguard_key_to_base64); + + match key + .as_ref() + .and_then(|k| plan.peers.iter().find(|p| &p.public_key == k)) + { + Some(peer) => tr + .set_tunnel_peer(&interface, peer) + .await + .map_err(|e| anyhow!("failed to configure peer on {interface}: {}", e))?, + // A tunnel that is disabled or has never presented a key has no + // peer to push. Removing whatever is there under its key is the + // same statement in the other direction. + None => { + if let Some(key) = &key { + tr.remove_tunnel_peer(&interface, key) + .await + .map_err(|e| anyhow!("failed to remove peer from {interface}: {}", e))?; + } + } + } + + tr.sync_tunnel_addresses(&interface, &plan.addresses) + .await + .map_err(|e| anyhow!("failed to configure addresses on {interface}: {}", e))?; + tr.sync_tunnel_routes(&interface, &plan.routes) + .await + .map_err(|e| anyhow!("failed to configure routes on {interface}: {}", e))?; Ok(()) } @@ -2569,6 +2788,12 @@ impl Worker { } => { self.remove_tunnel_interface(*router_id, interface).await?; } + WorkJob::ReconcileTunnelPeers { pool_id } => { + self.reconcile_tunnel_peers(*pool_id).await?; + } + WorkJob::SyncNodeTunnel { tunnel_id } => { + self.sync_node_tunnel(*tunnel_id).await?; + } WorkJob::DeleteVm { vm_id, reason, @@ -4970,6 +5195,276 @@ mod tests { .await?) } + /// An approved node with a backing host, holding a tunnel from `pool_id` + /// and one guest address. + async fn setup_node_tunnel(db: &Arc, pool_id: u64) -> Result { + use lnvps_db::{ + MarketplaceNode, MarketplaceNodeStatus, MarketplaceOperator, VmHost, VmHostKind, + }; + + let dbt: Arc = db.clone(); + let user_id = dbt.upsert_user(&[7u8; 32]).await?; + let operator_id = dbt + .insert_marketplace_operator(&MarketplaceOperator { + user_id, + enabled: true, + ..Default::default() + }) + .await?; + let node_id = dbt + .insert_marketplace_node(&MarketplaceNode { + operator_id, + name: "rack 1".to_string(), + status: MarketplaceNodeStatus::Approved, + ..Default::default() + }) + .await?; + let host_id = dbt + .create_host(&VmHost { + kind: VmHostKind::MarketplaceNode, + region_id: 1, + name: "node-host".to_string(), + ip: String::new(), + enabled: false, + marketplace_node_id: Some(node_id), + ..Default::default() + }) + .await?; + + let vm_id = { + let mut vms = db.vms.lock().await; + let id = vms.keys().max().copied().unwrap_or(0) + 1; + vms.insert( + id, + lnvps_db::Vm { + id, + host_id, + ..Default::default() + }, + ); + id + }; + dbt.insert_vm_ip_assignment(&lnvps_db::VmIpAssignment { + vm_id, + ip: "203.0.113.5".to_string(), + ..Default::default() + }) + .await?; + + let node = dbt.get_marketplace_node(node_id).await?; + let allocation = + crate::provisioner::allocate_node_tunnel(&dbt, &node, &[0x11u8; 32]).await?; + assert_eq!(allocation.tunnel.pool_id, Some(pool_id)); + Ok(allocation.tunnel) + } + + /// Configuring the interface is only half the job: the peers allocated from + /// the pool have to be on it, with an address on each link and a route for + /// each guest, or the node has a tunnel that carries nothing. + #[tokio::test] + async fn test_sync_tunnel_pool_realises_its_peers() -> Result<()> { + use crate::mocks::MockRouter; + + let db = Arc::new(MockDb::empty()); + let pool_id = setup_pool(&db, 51820).await?; + let tunnel = setup_node_tunnel(&db, pool_id).await?; + let mr = MockRouter::new(); + mr.clear().await; + + let worker = setup_worker(db.clone()).await?; + worker.sync_tunnel_pool(pool_id).await?; + + let interface = format!("wgln{pool_id}"); + let peers = mr.peers(&interface).await; + assert_eq!(peers.len(), 1); + assert_eq!( + peers[0].public_key, + lnvps_api_common::wireguard_key_to_base64(&[0x11u8; 32]) + ); + // The node's own address plus exactly the guest address assigned to it: + // this list is the anti-spoof boundary, not just a routing hint. + assert_eq!( + peers[0].allowed_ips, + vec!["10.66.0.2/32".to_string(), "203.0.113.5/32".to_string()] + ); + // One address for the pool, carrying the block's prefix: every node in + // it is on-link, so the route server does not carry an address per + // node on a single interface. + assert_eq!( + mr.interface_addresses(&interface).await, + vec!["10.66.0.1/24".to_string()] + ); + // AllowedIPs picks which peer a packet belongs to; it does not put the + // packet on the tunnel. Without this route the guest's return traffic + // is dropped as unroutable. + assert_eq!( + mr.interface_routes(&interface).await, + vec!["203.0.113.5/32".to_string()] + ); + assert_eq!(tunnel.pool_id, Some(pool_id)); + + mr.clear().await; + Ok(()) + } + + /// A peer that has vanished from a route server is drift to put back and + /// report, not an allocation to forget: forgetting it would hand the node's + /// addresses to somebody else while the node still uses them. + #[tokio::test] + async fn test_reconcile_tunnel_peers_repairs_and_reports_drift() -> Result<()> { + use crate::mocks::MockRouter; + use crate::router::{Router as _, WireguardPeer}; + + let db = Arc::new(MockDb::empty()); + let pool_id = setup_pool(&db, 51820).await?; + setup_node_tunnel(&db, pool_id).await?; + let mr = MockRouter::new(); + mr.clear().await; + let worker = setup_worker(db.clone()).await?; + worker.sync_tunnel_pool(pool_id).await?; + let interface = format!("wgln{pool_id}"); + + // Nothing changed: a working peer must not be rewritten on every poll, + // and `wg` reports allowed IPs in its own order. + let drift = worker.reconcile_tunnel_peers(pool_id).await?; + assert!(drift.is_empty(), "{drift}"); + + // The route server lost the peer (a reboot without persistence). + let tr = mr.tunnel().unwrap(); + let key = lnvps_api_common::wireguard_key_to_base64(&[0x11u8; 32]); + tr.remove_tunnel_peer(&interface, &key).await.unwrap(); + let drift = worker.reconcile_tunnel_peers(pool_id).await?; + assert_eq!(drift.missing, vec![key.clone()]); + assert_eq!(mr.peers(&interface).await.len(), 1, "not put back"); + + // A peer whose allowed IPs no longer match its allocation is carrying + // the wrong anti-spoof list, which is a security boundary, not cosmetic. + tr.set_tunnel_peer( + &interface, + &WireguardPeer { + public_key: key.clone(), + allowed_ips: vec!["0.0.0.0/0".to_string()], + ..Default::default() + }, + ) + .await + .unwrap(); + let drift = worker.reconcile_tunnel_peers(pool_id).await?; + assert_eq!(drift.changed, vec![key.clone()]); + assert_eq!( + mr.peers(&interface).await[0].allowed_ips, + vec!["10.66.0.2/32".to_string(), "203.0.113.5/32".to_string()] + ); + + // LNVPS owns `wgln*` outright, so a key no allocation accounts for is + // either a revoked node or somebody else's. Both are removed. + tr.set_tunnel_peer( + &interface, + &WireguardPeer { + public_key: "c3RyYXk=".to_string(), + ..Default::default() + }, + ) + .await + .unwrap(); + let drift = worker.reconcile_tunnel_peers(pool_id).await?; + assert_eq!(drift.unclaimed, vec!["c3RyYXk=".to_string()]); + assert_eq!(mr.peers(&interface).await.len(), 1); + + mr.clear().await; + Ok(()) + } + + /// Peers are configured *on* an interface. Creating it here would duplicate + /// the pool sync and hide the fact that it never ran. + #[tokio::test] + async fn test_reconcile_tunnel_peers_needs_the_interface() -> Result<()> { + use crate::mocks::MockRouter; + + let db = Arc::new(MockDb::empty()); + let pool_id = setup_pool(&db, 51820).await?; + let mr = MockRouter::new(); + mr.clear().await; + let worker = setup_worker(db.clone()).await?; + + let err = worker + .reconcile_tunnel_peers(pool_id) + .await + .expect_err("peers were reconciled onto an interface that is not there"); + assert!(format!("{err}").contains("SyncTunnelPool"), "{err}"); + + mr.clear().await; + Ok(()) + } + + /// One node getting an address must not wait behind a reconcile of every + /// other node on the same route server — and a tunnel that stops being + /// realisable takes its peer off the interface. + #[tokio::test] + async fn test_sync_node_tunnel_pushes_and_withdraws_one_peer() -> Result<()> { + use crate::mocks::MockRouter; + + let db = Arc::new(MockDb::empty()); + let pool_id = setup_pool(&db, 51820).await?; + let tunnel = setup_node_tunnel(&db, pool_id).await?; + let mr = MockRouter::new(); + mr.clear().await; + use crate::router::Router as _; + let worker = setup_worker(db.clone()).await?; + // The interface exists but has no peers, which is exactly the state + // right after a node asks for its tunnel. + worker.sync_tunnel_pool(pool_id).await?; + let interface = format!("wgln{pool_id}"); + let tr = mr.tunnel().unwrap(); + let key = lnvps_api_common::wireguard_key_to_base64(&[0x11u8; 32]); + tr.remove_tunnel_peer(&interface, &key).await.unwrap(); + + worker.sync_node_tunnel(tunnel.id).await?; + assert_eq!(mr.peers(&interface).await.len(), 1); + + // Disabling the allocation is a statement in the other direction: the + // peer comes off rather than being left behind carrying traffic. + let dbt: Arc = db.clone(); + dbt.update_tunnel(&lnvps_db::Tunnel { + enabled: false, + ..tunnel.clone() + }) + .await?; + worker.sync_node_tunnel(tunnel.id).await?; + assert!(mr.peers(&interface).await.is_empty()); + + mr.clear().await; + Ok(()) + } + + /// A tunnel allocated outside a pool has no interface to be configured on, + /// and inventing one would write a peer onto somebody else's tunnel. + #[tokio::test] + async fn test_sync_node_tunnel_without_a_pool_is_refused() -> Result<()> { + let db = Arc::new(MockDb::empty()); + let dbt: Arc = db.clone(); + let user_id = dbt.upsert_user(&[3u8; 32]).await?; + let tunnel_id = dbt + .insert_tunnel(&lnvps_db::Tunnel { + user_id, + name: "hand-made".to_string(), + enabled: true, + ..Default::default() + }) + .await?; + + let worker = setup_worker(db.clone()).await?; + let err = worker + .sync_node_tunnel(tunnel_id) + .await + .expect_err("a pool-less tunnel was pushed to an interface"); + assert!( + format!("{err}").contains("not allocated from a pool"), + "{err}" + ); + Ok(()) + } + /// A pool is not a description of an interface somebody configured by hand: /// syncing it creates the interface, with LNVPS's own key and port. #[tokio::test] diff --git a/lnvps_api_admin/src/admin/tunnel_pools.rs b/lnvps_api_admin/src/admin/tunnel_pools.rs index c8eb3908..591b6df3 100644 --- a/lnvps_api_admin/src/admin/tunnel_pools.rs +++ b/lnvps_api_admin/src/admin/tunnel_pools.rs @@ -46,10 +46,11 @@ pub fn router() -> Router { ) } -/// Point-to-point prefix lengths, matching the allocator. A pool's capacity is -/// counted in links, not addresses, because a link is what a node consumes. -const LINK_PREFIX_V4: u8 = 31; -const LINK_PREFIX_V6: u8 = 127; +/// A node consumes one address, matching the allocator: WireGuard needs no +/// gateway on the node's side, so a point-to-point link spent two addresses to +/// describe something that needs one. +const NODE_PREFIX_V4: u8 = 32; +const NODE_PREFIX_V6: u8 = 128; #[derive(Serialize, Debug)] pub struct AdminTunnelPoolInfo { @@ -165,21 +166,30 @@ pub struct UpdateTunnelPoolRequest { pub enabled: Option, } -/// How many point-to-point links `cidr` can supply, saturating: a /48 of IPv6 -/// holds more /127s than a `u64` can count, and the exact figure is not the +/// How many nodes `cidr` can supply addresses for, saturating: a /48 of IPv6 +/// holds more addresses than a `u64` can count, and the exact figure is not the /// point once it is that large. +/// +/// The addresses the route server reserves come off the top (see +/// `reserved_addresses` in the allocator), so this is what an admin can +/// actually place. fn link_capacity(cidr: Option<&str>) -> Option { let net: IpNetwork = cidr?.parse().ok()?; - let link_prefix = if net.is_ipv4() { - LINK_PREFIX_V4 + let node_prefix = if net.is_ipv4() { + NODE_PREFIX_V4 } else { - LINK_PREFIX_V6 + NODE_PREFIX_V6 }; - if net.prefix() > link_prefix { + if net.prefix() > node_prefix { return Some(0); } - let bits = link_prefix - net.prefix(); - Some(if bits >= 64 { u64::MAX } else { 1u64 << bits }) + let bits = node_prefix - net.prefix(); + let total = if bits >= 64 { u64::MAX } else { 1u64 << bits }; + // The route server holds the whole block on-link, so the block's own + // network address, the route server's address after it, and — on IPv4 — + // the broadcast address are not the pool's to hand out. + let reserved = if net.is_ipv4() { 3 } else { 2 }; + Some(total.saturating_sub(reserved)) } async fn pool_info( @@ -515,14 +525,12 @@ fn parse_block( if want_v4 { 4 } else { 6 } ))); } - let link_prefix = if want_v4 { - LINK_PREFIX_V4 - } else { - LINK_PREFIX_V6 - }; - if net.prefix() > link_prefix { + // Judged by what the block can actually place, not by its prefix: the + // route server's own address and the block's reserved ones come out of it, + // so a /31 that looks like it holds two nodes holds none. + if link_capacity(Some(value)).unwrap_or(0) == 0 { return Err(ApiError::bad_request(format!( - "{field} is smaller than a single /{link_prefix} link" + "{field} has no room for a node once the route server's own address is reserved" ))); } // Store the network address, so two pools written as `10.0.0.5/24` and @@ -634,9 +642,10 @@ mod tests { let info = pool_info(&db, pool).await.unwrap(); assert_eq!(info.router_name, "rs1"); assert_eq!(info.links_used, 0); - // A /24 holds 128 /31 links; the /64 holds far more, so the v4 block is - // what actually limits the pool. - assert_eq!(info.links_total, 128); + // A /24 places 253 nodes — 256 less its network address, the route + // server's own address and its broadcast address. The /64 holds far + // more, so the v4 block is what actually limits the pool. + assert_eq!(info.links_total, 253); } /// Capacity has to reflect what can be handed out, not the roomier of the @@ -656,21 +665,28 @@ mod tests { .unwrap(); let info = pool_info(&db, pool).await.unwrap(); - assert_eq!(info.links_total, 8, "the /28 limits the pool to 8 links"); + assert_eq!(info.links_total, 13, "the /28 limits the pool to 13 nodes"); } - /// A block exactly one link wide is not a rounding error, and one that - /// cannot hold a link reports no capacity rather than shifting negatively. + /// Capacity is what can actually be placed: the block's own reserved + /// addresses and the route server's are not a node's, and a block with no + /// room reports zero rather than shifting negatively. #[test] - fn capacity_counts_links_not_addresses() { - assert_eq!(link_capacity(Some("10.0.0.0/31")), Some(1)); - assert_eq!(link_capacity(Some("10.0.0.0/24")), Some(128)); - assert_eq!(link_capacity(Some("fd00::/127")), Some(1)); - // Saturated rather than overflowed: the exact number of /127s in a /48 - // is not a figure anyone needs. - assert_eq!(link_capacity(Some("fd00::/48")), Some(u64::MAX)); - // Smaller than one link. Creation refuses these, but an older row or a - // direct database edit must not panic the listing. + fn capacity_counts_what_can_be_placed() { + // 256 less the network, route server and broadcast addresses. + assert_eq!(link_capacity(Some("10.0.0.0/24")), Some(253)); + // A /30 is the smallest IPv4 block that can place anything at all. + assert_eq!(link_capacity(Some("10.0.0.0/30")), Some(1)); + assert_eq!(link_capacity(Some("10.0.0.0/31")), Some(0)); + // IPv6 has no broadcast address, so only two come off. + assert_eq!(link_capacity(Some("fd00::/127")), Some(0)); + assert_eq!(link_capacity(Some("fd00::/126")), Some(2)); + // Saturated rather than overflowed: the exact number of addresses in a + // /48 is not a figure anyone needs, and subtracting the reserved ones + // from a saturated count is noise at that scale. + assert_eq!(link_capacity(Some("fd00::/48")), Some(u64::MAX - 2)); + // Creation refuses these, but an older row or a direct database edit + // must not panic the listing. assert_eq!(link_capacity(Some("10.0.0.1/32")), Some(0)); assert_eq!(link_capacity(Some("not-a-cidr")), None); assert_eq!(link_capacity(None), None); @@ -679,13 +695,15 @@ mod tests { /// A v6 block in the v4 column would be handed out as an IPv4 link by an /// allocator that trusts the column. #[tokio::test] - async fn blocks_must_match_their_family_and_hold_a_link() { + async fn blocks_must_match_their_family_and_have_room() { let (db, router_id) = db().await; for (cidr4, cidr6) in [ (Some("fd00::/64"), None), (Some("not-a-cidr"), None), - // A /32 cannot hold a /31, and a /128 cannot hold a /127. + // Nothing left to place once the route server's own address and + // the block's reserved ones are taken out. (Some("10.0.0.1/32"), None), + (Some("10.0.0.0/31"), None), (None, Some("fd00::1/128")), (None, Some("10.0.0.0/24")), ] { diff --git a/lnvps_api_common/src/mock.rs b/lnvps_api_common/src/mock.rs index 9a1e689a..0e83db77 100644 --- a/lnvps_api_common/src/mock.rs +++ b/lnvps_api_common/src/mock.rs @@ -3617,6 +3617,17 @@ impl LNVpsDbBase for MockDb { .ok_or_else(|| DbError::Other(anyhow!("Marketplace node not found"))) } + async fn get_marketplace_node_by_tunnel( + &self, + tunnel_id: u64, + ) -> DbResult> { + let nodes = self.marketplace_nodes.lock().await; + Ok(nodes + .values() + .find(|n| n.tunnel_id == Some(tunnel_id)) + .cloned()) + } + async fn get_marketplace_node_by_line_item( &self, line_item_id: u64, diff --git a/lnvps_api_common/src/work/mod.rs b/lnvps_api_common/src/work/mod.rs index eab53247..f0fb6945 100644 --- a/lnvps_api_common/src/work/mod.rs +++ b/lnvps_api_common/src/work/mod.rs @@ -240,6 +240,20 @@ pub enum WorkJob { /// *after* the pool row is gone — the alternative is deleting the row and /// leaving a configured interface behind with no record that it exists. RemoveTunnelInterface { router_id: u64, interface: String }, + /// Reconcile the peers, addresses and routes on a tunnel pool's interface + /// against the tunnels allocated from it. + /// + /// Separate from [`SyncTunnelPool`](Self::SyncTunnelPool) because that + /// re-applies the interface itself, which on Linux means recreating it and + /// dropping every peer. This one touches only what has actually drifted, + /// which is what makes it safe to run on a schedule. + ReconcileTunnelPeers { pool_id: u64 }, + /// Push one node's peer onto its route server. + /// + /// The fast path for "this node just got an address": a full reconcile of + /// the pool would work, but a node waiting on its first guest should not + /// wait for every other node on the route server to be checked first. + SyncNodeTunnel { tunnel_id: u64 }, /// Re-apply forward + reverse DNS records for every IP assignment in a range. /// /// Used after changing a range's DNS server configuration (e.g. switching @@ -347,6 +361,8 @@ impl fmt::Display for WorkJob { WorkJob::ToggleTunnel { .. } => write!(f, "ToggleTunnel"), WorkJob::SyncTunnelPool { .. } => write!(f, "SyncTunnelPool"), WorkJob::RemoveTunnelInterface { .. } => write!(f, "RemoveTunnelInterface"), + WorkJob::ReconcileTunnelPeers { .. } => write!(f, "ReconcileTunnelPeers"), + WorkJob::SyncNodeTunnel { .. } => write!(f, "SyncNodeTunnel"), WorkJob::PatchIpRangeDns { .. } => write!(f, "PatchIpRangeDns"), } } @@ -399,5 +415,13 @@ mod tests { .to_string(), "RemoveTunnelInterface" ); + assert_eq!( + WorkJob::ReconcileTunnelPeers { pool_id: 2 }.to_string(), + "ReconcileTunnelPeers" + ); + assert_eq!( + WorkJob::SyncNodeTunnel { tunnel_id: 5 }.to_string(), + "SyncNodeTunnel" + ); } } diff --git a/lnvps_db/src/lib.rs b/lnvps_db/src/lib.rs index 16c1aea6..3ef2ed8b 100644 --- a/lnvps_db/src/lib.rs +++ b/lnvps_db/src/lib.rs @@ -1118,6 +1118,17 @@ pub trait LNVpsDbBase: Send + Sync { fingerprint: &[u8], ) -> DbResult; + /// Get the marketplace node terminating `tunnel_id`, if any. + /// + /// The tunnel table deliberately records nothing about what a tunnel is + /// *for*, so realising a peer has to come back this way to find whose guest + /// addresses belong behind it. `uk_marketplace_node_tunnel` makes the + /// answer single-valued. + async fn get_marketplace_node_by_tunnel( + &self, + tunnel_id: u64, + ) -> DbResult>; + /// Get the marketplace node whose listing fee is billed by `line_item_id`. /// /// The back-reference direction matches `vm.subscription_line_item_id`: the diff --git a/lnvps_db/src/mysql.rs b/lnvps_db/src/mysql.rs index 9f20fde3..d3276f12 100644 --- a/lnvps_db/src/mysql.rs +++ b/lnvps_db/src/mysql.rs @@ -4173,6 +4173,18 @@ impl LNVpsDbBase for LNVpsDbMysql { ) } + async fn get_marketplace_node_by_tunnel( + &self, + tunnel_id: u64, + ) -> DbResult> { + Ok( + sqlx::query_as("SELECT * FROM marketplace_node WHERE tunnel_id = ?") + .bind(tunnel_id) + .fetch_optional(&self.db) + .await?, + ) + } + async fn get_marketplace_node_by_line_item( &self, line_item_id: u64, diff --git a/work/marketplace.md b/work/marketplace.md index 4a233a4d..83ea27b3 100644 --- a/work/marketplace.md +++ b/work/marketplace.md @@ -742,9 +742,15 @@ What the build settled beyond the plan: `tunnel_pool (id, router_id)` is enforced by MariaDB, verified against a real server: a tunnel claiming a pool on another router is rejected by the database, and a NULL `pool_id` skips the constraint, which is exactly the pool-less case. The mock mirrors it. -- **The peer's address is the network address with the low bit set**, not plus one. On a /31 or - /127 the network address always has that bit clear, so there is no overflow case and no - arithmetic error to invent — the last link in a block is still a link. +- **A node takes one address, not a link** (revised during 4b; 4a shipped /31s and /127s). + WireGuard is layer 3 and point-to-point, with no ARP and no on-link requirement, so the node + needs no gateway of its own — `ip route add default dev wg0` is enough. A /31 therefore spent + two addresses describing something that needs one, and worse, forced the route server to hold + one address per node on a single interface: a /16 pool with a thousand nodes meant a thousand + addresses on `wgln`, re-parsed out of `ip addr show` on every reconcile. The route server + now holds **one** address per pool, carrying the block's own prefix so every node in it is + on-link. The block's network address, that address, and (on IPv4) the broadcast address are + reserved, so a /24 places 253 nodes. - **A dual-stack pool's capacity is the smaller block's**, because a link of each family is handed out together. Reporting the roomier one would promise capacity that cannot be allocated. @@ -776,7 +782,48 @@ the allocator; the admin pool handlers *are* covered end to end. an allocation is not a working tunnel. - Admin CRUD for pools, with utilisation. -#### 4b — Route-server realisation + drift (M/L) ⬅ NEXT +#### 4b — Route-server realisation + drift (M/L) ✅ DONE + +What the build settled beyond the plan: +- **Peers are pushed one at a time, not through the interface.** `update_tunnel` recreates the + interface on the Linux backend and takes every peer with it, so one node getting a guest + address would cut every other node on the route server. `TunnelRouter` grew `set_tunnel_peer` + / `remove_tunnel_peer`, which are `wg set peer` — additive, idempotent, and leaving the rest + of the interface alone. +- **`AllowedIPs` narrows as well as widens**, which is what makes it usable as the anti-spoof + boundary rather than a routing hint: `wg set peer allowed-ips` *replaces* the list, so a guest + address that was released stops being accepted from that node on the next reconcile. +- **AllowedIPs is not a route.** It picks which peer a packet already headed down the tunnel + belongs to; without `ip route` the guest's return traffic reaches the route server and is + dropped as unroutable. Hence `sync_tunnel_routes` alongside `sync_tunnel_addresses`, both + declarative — the caller knows the desired set, working out the difference is the backend's + job. +- **A sync must not touch what the kernel owns.** The IPv6 link-local address and the /31 link + route are put there by the kernel; a reconcile that deleted everything it did not add would + fight the kernel on every poll. Both are excluded explicitly. +- **Both address families have to be queried separately.** `ip route show` is IPv4 only, so a v6 + guest prefix would look absent on every sync and be re-added forever. +- **Drift is reported, not just repaired.** `missing`, `changed` and `unclaimed` are kept apart + because they mean different things: a peer that is gone was configured and vanished, a changed + one is carrying the wrong anti-spoof list, and an unclaimed one is a key on an LNVPS interface + that no allocation accounts for. Unclaimed peers are **removed** — `wgln*` is ours outright. +- **Allowed IPs are compared as a set.** `wg` reports them in its own order; treating that as a + difference would rewrite a working peer's security boundary on every single poll. +- **Reconcile refuses to create the interface.** Peers are configured *on* an interface, and + creating it here would duplicate `SyncTunnelPool` and hide the fact that it never ran. +- **Guest addressing is corrected by the existing router poll**, not by wiring a route-server + call into the VM provisioning path. `SyncNodeTunnel` exists for promptness when a node asks + for its tunnel; correctness does not depend on it firing. +- **A Mikrotik route server refuses peer operations rather than ignoring them.** The four new + methods default to an error, so a pool put on a backend that cannot carry it fails loudly + instead of accepting the pool and configuring nothing. + +Testing note: `LinuxSshRouter` gained a `#[cfg(test)]` command hook. These methods run commands +as root on somebody else's route server, so what is worth asserting is the exact command issued +— which needs the transport replaced, not mocked around. The rest of that file remains untested +for want of a real box, as before. + +#### 4b — original scope - Push the peer to the route server through the existing `TunnelRouter`: `AllowedIPs` is the node's inner addresses **plus** the guest IPs assigned to it, which is also the anti-spoof boundary. @@ -784,7 +831,7 @@ the allocator; the admin pool handlers *are* covered end to end. from a router is drift to report, not an allocation to forget. - Route the guest prefixes at the route server towards the peer. -#### 4c — Node data plane + health gate (L) +#### 4c — Node data plane + health gate (L) ⬅ NEXT - Node side: `wg0` + `br-lnvps`, default route into the tunnel, anti-spoof and anti-LAN-access rules, MTU/MSS clamp. - Health gate: the host is only enabled after an end-to-end reachability probe from the route