From e4bbfa292a4e41f3fa0fc06ab5269ae87e8bb2f8 Mon Sep 17 00:00:00 2001 From: v0l Date: Thu, 6 Aug 2026 17:11:21 +0100 Subject: [PATCH 1/3] feat(marketplace): the node configures its own data plane MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A node had a tunnel allocated to it and no way to use one. This gives it the document that says what its network should be, and the code that makes the machine match. `GET /api/v1/node/dataplane` returns the whole thing at once: the tunnel, the bridge guests are placed on, the gateway addresses the node must answer for, and the guests assigned to it. One call rather than three because the node applies them together or not at all — a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list the firewall increment will enforce. **The daemon configures the machine itself**, with `ip` and `wg`, rather than writing files for something else to read. A marketplace node runs on hardware LNVPS does not own: a data plane that depends on the operator having wired it up correctly is one whose mistakes surface as a customer's VM having no network. Everything is stated declaratively — `ip addr replace`, `ip route replace`, `wg set` — so a node that is already right is not disturbed and one that has drifted is corrected without being torn down. Three things the shape of the network forced: - **A guest's gateway belongs to its range, not to this node.** The guest is configured with the range's gateway and believes it is on-link, so the node holds that address on the bridge as a *host* address and answers for it with proxy ARP/NDP. Holding the whole range instead would make the node believe every other node's guests were local, and their traffic would vanish into the bridge rather than going up the tunnel. - **The bridge takes the tunnel's MTU.** A guest sending 1500 bytes into a 1420-byte tunnel gets a connection that opens and then hangs on the first large transfer, which is the worst failure shape available. - **The data plane is applied before the listener binds.** The control API binds an address of the tunnel interface, and on a fresh machine that interface does not exist until this has run. A failure to fetch is a warning rather than fatal: a node already up from a previous run must keep serving through an LNVPS outage, or one API blip takes the whole fleet dark. Two hygiene points that are not incidental. A peer that is not the route server is removed from `wg0` — most likely a stale key from a re-key, otherwise still able to send traffic the node treats as LNVPS's. And routes for departed guests are swept, because a released address goes straight back in the pool and may already be somebody else's; the bridge's own gateways are excluded from that sweep so tidying up after a guest cannot take the bridge's addressing with it. The node's WireGuard key is generated in-process rather than by shelling out to `wg genkey`, so a missing `wg` fails when the interface is configured, with that error. It is written `0600` with the mode set *before* the key exists, and reaches `wg` as a path, never an argument: arguments are visible in `ps` to every user on the machine, and these machines usually have more than one login. `/api/v1/status` now reports the observed data plane, queried on demand rather than cached — a cached answer says the tunnel was up once, which is exactly what the health gate must not accept. A tunnel that has never handshaken is reported as configured but not working, because `wg0` comes up perfectly happily with a peer that never answers. `lnvps-node dataplane show|apply|observe` exposes the same thing to an operator; `observe` deliberately needs no credential, because "what does this machine actually have?" is the question asked when something is already broken. --- API_CHANGELOG.md | 2 + Cargo.lock | 4 + lnvps_api/src/api/marketplace.rs | 62 ++ lnvps_api/src/provisioner/tunnel.rs | 166 ++++++ lnvps_node/Cargo.toml | 11 + lnvps_node/config.example.yaml | 5 + lnvps_node/src/api.rs | 280 +++++++++ lnvps_node/src/control.rs | 47 +- lnvps_node/src/lib.rs | 7 + lnvps_node/src/main.rs | 134 ++++- lnvps_node/src/net.rs | 843 ++++++++++++++++++++++++++++ lnvps_node/src/wgkey.rs | 292 ++++++++++ work/marketplace.md | 91 ++- 13 files changed, 1936 insertions(+), 8 deletions(-) create mode 100644 lnvps_node/src/api.rs create mode 100644 lnvps_node/src/net.rs create mode 100644 lnvps_node/src/wgkey.rs diff --git a/API_CHANGELOG.md b/API_CHANGELOG.md index 23886f5e..79be4603 100644 --- a/API_CHANGELOG.md +++ b/API_CHANGELOG.md @@ -56,6 +56,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added +- **A node can fetch its whole data plane in one document** — `GET /api/v1/node/dataplane` (node token) returns the tunnel it already gets from `/node/tunnel`, plus the bridge guests are placed on, the gateway addresses the node must answer for, and the guests assigned to it (address, gateway, MAC). One call rather than three because the node applies these together or not at all: a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list — an address not in it is not that node's to send from. The gateway is the one the guest was actually configured with, taken from its IP range: it belongs to the range rather than to the node, and the guest believes it is on-link, so the node has to answer for it rather than invent one. + - **A node takes one address, not a point-to-point link** — `address4`/`address6` in the tunnel response are now a `/32` and a `/128`, and `gateway4`/`gateway6` are one address shared by every node on the pool rather than a per-node link address. WireGuard is layer 3 and point-to-point: the node needs no gateway on its own side (`ip route add default dev wg0` suffices), so a `/31` spent two addresses describing something that needs one — and forced the route server to carry one address per node on a single interface, thousands of them on a /16 pool. Pool capacity is reported accordingly: a /24 places 253 nodes (256 less the block's network address, the route server's address after it, and the broadcast address) where it previously reported 128 links. - **A marketplace node's tunnel is now actually configured on the route server** — allocating a tunnel (`POST /api/v1/node/tunnel`) previously wrote down addresses and a key and stopped there; the node's peer is now pushed to the route server, so the tunnel carries traffic instead of only existing on paper. The peer's `AllowedIPs` is the node's own inner addresses plus exactly the guest addresses LNVPS assigned to it, which is also the anti-spoof boundary: WireGuard drops an inbound packet whose source is not on that list, so a node cannot source traffic as another node's customer. The route server takes an address on each point-to-point link, and each guest address is routed down the interface — `AllowedIPs` decides which *peer* a packet already headed for the tunnel belongs to, it does not put the packet there. Peers are also reconciled on the routine router poll, so an address assigned to a guest since the last push, a peer wiped by a route-server reboot, and a stale key nobody removed are all corrected without an admin doing anything; the reconcile reports what had drifted rather than quietly fixing it. The backing host stays **disabled** — a configured peer is still not a proven path, and the health gate that enables it comes with the node-side data plane. diff --git a/Cargo.lock b/Cargo.lock index 36ac2478..244d3a0f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3611,10 +3611,13 @@ dependencies = [ "clap", "config", "env_logger", + "hex", + "http", "if-addrs", "lnvps_host_util", "log 0.4.32", "nostr 0.44.3", + "rand 0.9.4", "rcgen", "reqwest 0.12.28", "rustls", @@ -3624,6 +3627,7 @@ dependencies = [ "tokio", "tokio-rustls", "tower", + "x25519-dalek", ] [[package]] diff --git a/lnvps_api/src/api/marketplace.rs b/lnvps_api/src/api/marketplace.rs index ab971f5e..171126b3 100644 --- a/lnvps_api/src/api/marketplace.rs +++ b/lnvps_api/src/api/marketplace.rs @@ -50,6 +50,7 @@ pub fn router() -> Router { "/api/v1/node/tunnel", get(v1_node_get_tunnel).post(v1_node_request_tunnel), ) + .route("/api/v1/node/dataplane", get(v1_node_dataplane)) } /// A node as its operator sees it. @@ -609,6 +610,67 @@ async fn v1_node_get_tunnel( ApiData::ok(allocation.into()) } +/// Everything the node's data plane should look like, in one document. +#[derive(Serialize, Debug)] +pub struct ApiNodeDataPlane { + pub tunnel: ApiNodeTunnel, + /// The bridge guests are placed on. LNVPS decides the name so every node is + /// the same shape. + pub bridge: String, + /// Gateway addresses this node must answer for on the bridge. They belong + /// to the ranges the guests were addressed from, and the guests believe + /// they are on-link. + pub gateways: Vec, + /// The guests placed here. Also the anti-spoof list: an address that is not + /// in it is not this node's to send from. + pub guests: Vec, +} + +#[derive(Serialize, Debug)] +pub struct ApiNodeGuest { + /// Host prefix (`203.0.113.5/32`), so v4 and v6 read the same way. + pub address: String, + /// The gateway this guest was configured with. + pub gateway: String, + /// The guest's MAC, when recorded. + pub mac: Option, +} + +impl From for ApiNodeDataPlane { + fn from(d: crate::provisioner::NodeDataPlane) -> Self { + Self { + gateways: d.gateways(), + bridge: d.bridge.clone(), + guests: d + .guests + .iter() + .map(|g| ApiNodeGuest { + address: g.address.clone(), + gateway: g.gateway.clone(), + mac: g.mac.clone(), + }) + .collect(), + tunnel: d.tunnel.into(), + } + } +} + +/// The node's whole desired data plane. +/// +/// One call rather than three, because the node applies these together or not +/// at all: a bridge with no tunnel carries nothing, and a tunnel with no guest +/// routes carries nothing back. +async fn v1_node_dataplane( + auth: NodeAuth, + State(this): State, +) -> ApiResult { + let plane = crate::provisioner::node_dataplane(&this.db, &auth.node) + .await + .map_err(|e| ApiError::new(e.to_string()))? + .ok_or_else(|| ApiError::not_found("This node has no tunnel allocated yet"))?; + ApiData::ok(plane.into()) +} + /// What a node is told about itself. /// /// The daemon uses this to confirm its token works and to see whether it has diff --git a/lnvps_api/src/provisioner/tunnel.rs b/lnvps_api/src/provisioner/tunnel.rs index 46fdec88..c39d1d4d 100644 --- a/lnvps_api/src/provisioner/tunnel.rs +++ b/lnvps_api/src/provisioner/tunnel.rs @@ -407,6 +407,109 @@ pub async fn plan_pool(db: &Arc, pool: &TunnelPool) -> Result, +} + +/// One address assigned to a guest on a node. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GuestAddress { + /// The guest's address as a host prefix (`203.0.113.5/32`). + pub address: String, + /// The gateway the guest is configured with. + /// + /// It belongs to the range, not to this node, and the guest believes it is + /// on-link — so the node has to answer for it on the bridge. Sent per + /// address rather than per node because one node can hold guests from + /// several ranges. + pub gateway: String, + /// The guest's MAC, when one is recorded. Lets the node bind an address to + /// a port rather than trusting whatever claims it. + pub mac: Option, +} + +impl NodeDataPlane { + /// The gateway addresses this node has to answer for, deduplicated. + pub fn gateways(&self) -> Vec { + let mut out: Vec = self.guests.iter().map(|g| g.gateway.clone()).collect(); + out.sort(); + out.dedup(); + out + } +} + +/// Assemble the desired data plane for `node`. +pub async fn node_dataplane( + db: &Arc, + node: &MarketplaceNode, +) -> Result> { + let Some(tunnel) = get_node_tunnel(db, node).await? else { + return Ok(None); + }; + Ok(Some(NodeDataPlane { + guests: node_guests(db, node).await?, + tunnel, + bridge: NODE_BRIDGE.to_string(), + })) +} + +/// The guests placed on `node`, with what each needs to be reachable. +pub async fn node_guests( + db: &Arc, + node: &MarketplaceNode, +) -> Result> { + let Some(host) = db.get_marketplace_node_host(node.id).await? else { + return Ok(vec![]); + }; + let mut out = Vec::new(); + for vm in db.list_vms_on_host(host.id).await? { + if vm.deleted { + continue; + } + for ip in db.list_vm_ip_assignments(vm.id).await? { + if ip.deleted { + continue; + } + let Some(address) = host_address(Some(&ip.ip)) else { + continue; + }; + // The range is what says which gateway the guest was handed; a + // node inventing one would answer for an address the guest never + // uses. Normalised to a bare address because a stored gateway may + // carry the range's prefix, and the node holds it as a host + // address on the bridge either way. + let range = db.get_ip_range(ip.ip_range_id).await?; + let gateway = lnvps_api_common::parse_gateway(&range.gateway) + .map(|g| g.ip().to_string()) + .unwrap_or(range.gateway); + out.push(GuestAddress { + address, + gateway, + mac: Some(vm.mac_address.clone()).filter(|m| !m.is_empty()), + }); + } + } + out.sort_by(|a, b| a.address.cmp(&b.address)); + Ok(out) +} + /// The public addresses assigned to the guests running on `tunnel`'s node. /// /// Empty for a tunnel that is not a marketplace node's, or a node with no @@ -1066,6 +1169,67 @@ mod tests { let _ = mock; } + /// The document is what the node acts on, so it has to carry everything the + /// node cannot work out for itself: the gateway a guest was configured with + /// belongs to the range, not to the node, and a node inventing one would + /// answer for an address no guest uses. + #[tokio::test] + async fn the_data_plane_document_describes_the_whole_node() { + let (db, mock, node, _) = fixture().await; + allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let node = db.get_marketplace_node(node.id).await.unwrap(); + let host = db + .get_marketplace_node_host(node.id) + .await + .unwrap() + .unwrap(); + add_guest(&db, &mock, host.id, &["203.0.113.5", "2001:db8::5"]).await; + + let plane = node_dataplane(&db, &node).await.unwrap().unwrap(); + assert_eq!(plane.bridge, NODE_BRIDGE); + assert_eq!( + plane.tunnel.tunnel.address4.as_deref(), + Some("10.66.0.2/32") + ); + assert_eq!( + plane + .guests + .iter() + .map(|g| g.address.as_str()) + .collect::>(), + ["2001:db8::5/128", "203.0.113.5/32"] + ); + // Bare, even though the range stores it with a prefix: the node holds + // it as a host address on the bridge. + assert_eq!(plane.guests[0].gateway, "10.0.0.1"); + assert_eq!(plane.guests[0].mac.as_deref(), Some("aa:bb:cc:dd:ee:ff")); + // Deduplicated: two guests from one range give the node one address to + // answer for, not two identical ones. + assert_eq!(plane.gateways(), vec!["10.0.0.1".to_string()]); + } + + /// A node with no tunnel has no data plane to describe — saying so beats + /// returning a document with an empty tunnel that a node would apply. + #[tokio::test] + async fn a_node_without_a_tunnel_has_no_document() { + let (db, _mock, node, _) = fixture().await; + assert!(node_dataplane(&db, &node).await.unwrap().is_none()); + } + + /// A node with no guests yet is still configured: it is realised before it + /// has customers, so it can be given some. + #[tokio::test] + async fn a_node_with_no_guests_still_has_a_document() { + let (db, _mock, node, _) = fixture().await; + allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let node = db.get_marketplace_node(node.id).await.unwrap(); + + let plane = node_dataplane(&db, &node).await.unwrap().unwrap(); + assert!(plane.guests.is_empty()); + assert!(plane.gateways().is_empty()); + assert_eq!(plane.bridge, NODE_BRIDGE); + } + /// Give `host` a VM holding `ips`. Written through the mock's maps because /// a VM needs a template, an image and a disk that this test does not care /// about. @@ -1078,6 +1242,7 @@ mod tests { lnvps_db::Vm { id, host_id, + mac_address: "aa:bb:cc:dd:ee:ff".to_string(), ..Default::default() }, ); @@ -1087,6 +1252,7 @@ mod tests { db.insert_vm_ip_assignment(&lnvps_db::VmIpAssignment { vm_id, ip: ip.to_string(), + ip_range_id: 1, ..Default::default() }) .await diff --git a/lnvps_node/Cargo.toml b/lnvps_node/Cargo.toml index a9710e54..081d5e19 100644 --- a/lnvps_node/Cargo.toml +++ b/lnvps_node/Cargo.toml @@ -15,6 +15,15 @@ path = "src/main.rs" lnvps_host_util = { path = "../lnvps_host_util", default-features = false } nostr = { version = "0.44", default-features = false, features = ["std"] } +# The node's own WireGuard key, generated here so LNVPS never holds the private +# half. Generated in-process rather than by shelling out to `wg genkey`, and +# without depending on lnvps_api_common, which would pull the database, axum and +# the payment stack onto somebody else's hardware. +x25519-dalek = { version = "3.0.0", features = ["static_secrets"] } +rand = "0.9" +hex.workspace = true +# Outbound calls to LNVPS: the node fetches the data plane it should be running. +reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] } # Self-signed TLS identity, pinned by LNVPS at registration. Same crate and # approach as lnvps_fw's control API. rcgen = "0.13" @@ -41,6 +50,8 @@ config.workspace = true [dev-dependencies] tempfile = "3" +# Builds responses for the API-client tests without a live server. +http = "1" # Drives the router directly in tests, so authentication is exercised without # binding a socket or terminating TLS. tower = { version = "0.5", features = ["util"] } diff --git a/lnvps_node/config.example.yaml b/lnvps_node/config.example.yaml index ceba0c0c..07182eb0 100644 --- a/lnvps_node/config.example.yaml +++ b/lnvps_node/config.example.yaml @@ -25,6 +25,11 @@ heartbeat-secs: 60 # Inbound control API. Omit this section until the node is paired — without it # the daemon has nothing to serve and will say so. +# +# The daemon brings the tunnel up itself before binding, so `listen` is an +# address that exists because of what the daemon did, not something to configure +# by hand: it is the address LNVPS allocated, which `lnvps-node dataplane show` +# prints. control: # Must be an address of tunnel-interface below, and never a wildcard. The # control API can start and stop other people's virtual machines, so it is diff --git a/lnvps_node/src/api.rs b/lnvps_node/src/api.rs new file mode 100644 index 00000000..9fbc9cc2 --- /dev/null +++ b/lnvps_node/src/api.rs @@ -0,0 +1,280 @@ +//! Outbound calls to LNVPS. +//! +//! This is the only direction that works before a tunnel exists, which is why +//! the node fetches its data plane rather than being handed it: the control API +//! LNVPS would push it over is reachable *through* the tunnel it describes. +//! +//! Every call carries the node's own token. A node authenticates as itself, +//! never as its operator, so a compromised machine costs the operator that node +//! and nothing else. + +use anyhow::{Context, Result, bail}; +use serde::Deserialize; + +use crate::credential::Credential; +use crate::net::DesiredDataPlane; + +/// A client for the LNVPS node API. +pub struct LnvpsApi { + base_url: String, + authorization: String, + http: reqwest::Client, +} + +/// The envelope every LNVPS response comes in. +#[derive(Deserialize)] +struct ApiResponse { + data: Option, + error: Option, +} + +impl LnvpsApi { + pub fn new(base_url: &str, credential: &Credential) -> Result { + Ok(Self { + base_url: base_url.trim_end_matches('/').to_string(), + authorization: credential.authorization_header(), + http: reqwest::Client::builder() + // A node that cannot reach LNVPS must find out in seconds. The + // default is no timeout at all, which turns an unreachable API + // into a daemon that never finishes starting. + .timeout(std::time::Duration::from_secs(30)) + .build() + .context("Cannot build an HTTP client")?, + }) + } + + /// Present the node's public key and receive its tunnel allocation. + /// + /// Idempotent at the far end: a node that asks twice gets the allocation it + /// already has, and one presenting a new key is re-pinned in place. + pub async fn request_tunnel(&self, public_key: &[u8; 32]) -> Result<()> { + let url = format!("{}/api/v1/node/tunnel", self.base_url); + let response = self + .http + .post(&url) + .header("Authorization", &self.authorization) + .json(&serde_json::json!({ "public_key": hex::encode(public_key) })) + .send() + .await + .with_context(|| format!("Cannot reach LNVPS at {url}"))?; + let _: serde_json::Value = decode(response, &url).await?; + Ok(()) + } + + /// Fetch the data plane this node should be running. + pub async fn dataplane(&self) -> Result { + let url = format!("{}/api/v1/node/dataplane", self.base_url); + let response = self + .http + .get(&url) + .header("Authorization", &self.authorization) + .send() + .await + .with_context(|| format!("Cannot reach LNVPS at {url}"))?; + decode(response, &url).await + } +} + +/// Unwrap an LNVPS response, preferring its own error message to the status. +/// +/// The API reports failure in the body as often as in the status, and "500 +/// Internal Server Error" tells an operator nothing they can act on where +/// "This node has no tunnel allocated yet" tells them exactly what to do. +async fn decode( + response: reqwest::Response, + url: &str, +) -> Result { + let status = response.status(); + let body = response + .text() + .await + .with_context(|| format!("Cannot read the response from {url}"))?; + + match serde_json::from_str::>(&body) { + Ok(parsed) => { + if let Some(error) = parsed.error { + bail!("{url}: {error}"); + } + match parsed.data { + Some(data) => Ok(data), + None => bail!("{url}: response carried neither data nor an error"), + } + } + // Not an LNVPS envelope at all: a proxy error page, or the wrong URL. + // Reporting the status and a bounded slice of the body beats a serde + // message about a missing field. + Err(e) => bail!( + "{url}: {status}, and the response is not an LNVPS response ({e}): {}", + body.chars().take(200).collect::() + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Mutex; + + #[derive(Deserialize, Debug, PartialEq)] + struct Thing { + value: u32, + } + + fn response(status: u16, body: &str) -> reqwest::Response { + reqwest::Response::from( + http::Response::builder() + .status(status) + .body(body.to_string()) + .unwrap(), + ) + } + + /// The ordinary case: the envelope's data is what the caller wanted. + #[tokio::test] + async fn a_successful_response_is_unwrapped() { + let got: Thing = decode(response(200, r#"{"data":{"value":7}}"#), "u") + .await + .unwrap(); + assert_eq!(got, Thing { value: 7 }); + } + + /// The API reports failure in the body as often as in the status, and its + /// message is the one an operator can act on. + #[tokio::test] + async fn the_apis_own_error_is_preferred_to_the_status() { + let err = decode::( + response(404, r#"{"error":"This node has no tunnel allocated yet"}"#), + "u", + ) + .await + .unwrap_err(); + assert!(format!("{err}").contains("no tunnel allocated"), "{err}"); + } + + /// A proxy error page is not an LNVPS response; saying so beats a serde + /// message about a missing field. + #[tokio::test] + async fn a_non_lnvps_response_is_reported_with_its_status() { + let err = decode::(response(502, "bad gateway"), "u") + .await + .unwrap_err(); + let text = format!("{err}"); + assert!(text.contains("502"), "{text}"); + assert!(text.contains("bad gateway"), "{text}"); + } + + /// An envelope carrying neither half is a bug at the far end, and a node + /// that treated it as success would apply an empty data plane. + #[tokio::test] + async fn an_empty_envelope_is_an_error() { + assert!(decode::(response(200, "{}"), "u").await.is_err()); + } + + fn credential() -> Credential { + Credential::parse( + "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuaWQiOjd9.c2ln", + std::path::Path::new("/etc/lnvps-node/token"), + ) + .unwrap() + } + + /// A real server on a real socket, because what is worth proving is the + /// request that goes out: the node's token on every call, the public key + /// presented before the document is asked for, and the document parsed as + /// the node will actually apply it. + #[tokio::test] + async fn the_node_presents_its_key_and_fetches_its_document() { + use axum::routing::{get, post}; + use std::sync::Arc; + + let seen: Arc>> = Arc::new(Mutex::new(Vec::new())); + let recorder = seen.clone(); + let app = axum::Router::new() + .route( + "/api/v1/node/tunnel", + post(|headers: axum::http::HeaderMap, body: String| async move { + recorder.lock().unwrap().push(( + "POST /api/v1/node/tunnel".to_string(), + headers + .get("authorization") + .and_then(|h| h.to_str().ok()) + .unwrap_or_default() + .to_string(), + body, + )); + axum::Json(serde_json::json!({ "data": {} })) + }), + ) + .route( + "/api/v1/node/dataplane", + get(|| async { + axum::Json(serde_json::json!({ "data": { + "tunnel": { + "address4": "10.66.0.2/32", + "address6": null, + "gateway4": "10.66.0.1", + "gateway6": null, + "server_public_key": "ab".repeat(32), + "endpoint": "rs1.example:51820", + "keepalive": 25, + "mtu": 1420 + }, + "bridge": "br-lnvps", + "gateways": ["203.0.113.1"], + "guests": [ + {"address": "203.0.113.5/32", "gateway": "203.0.113.1", "mac": null} + ] + }})) + }), + ); + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); + + let api = LnvpsApi::new(&format!("http://{addr}"), &credential()).unwrap(); + api.request_tunnel(&[0x11; 32]).await.unwrap(); + let plane = api.dataplane().await.unwrap(); + + let calls = seen.lock().unwrap().clone(); + assert_eq!(calls.len(), 1); + assert!(calls[0].1.starts_with("Bearer "), "{:?}", calls[0]); + // Hex, as the rest of the node API states keys. + assert!( + calls[0].2.contains(&hex::encode([0x11u8; 32])), + "{:?}", + calls[0] + ); + + assert_eq!(plane.bridge, "br-lnvps"); + assert_eq!(plane.tunnel.mtu, 1420); + assert_eq!(plane.guests.len(), 1); + assert_eq!(plane.gateways, vec!["203.0.113.1".to_string()]); + } + + /// An unreachable API is a message naming the URL, not a bare connection + /// error: an operator reading a node's log has to know what it was dialling. + #[tokio::test] + async fn an_unreachable_api_names_what_it_was_dialling() { + // Port 1 on localhost: nothing listens there, and nothing can. + let api = LnvpsApi::new("http://127.0.0.1:1", &credential()).unwrap(); + let err = api.dataplane().await.unwrap_err(); + assert!(format!("{err:#}").contains("127.0.0.1:1"), "{err:#}"); + let err = api.request_tunnel(&[0x11; 32]).await.unwrap_err(); + assert!(format!("{err:#}").contains("node/tunnel"), "{err:#}"); + } + + /// The base URL is normalised so a trailing slash in the config file does + /// not produce `//api/v1/...`, which some proxies redirect and others 404. + #[test] + fn the_base_url_is_normalised() { + let credential = Credential::parse( + "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuaWQiOjd9.c2ln", + std::path::Path::new("/etc/lnvps-node/token"), + ) + .unwrap(); + let api = LnvpsApi::new("https://api.lnvps.net/", &credential).unwrap(); + assert_eq!(api.base_url, "https://api.lnvps.net"); + assert!(api.authorization.starts_with("Bearer ")); + } +} diff --git a/lnvps_node/src/control.rs b/lnvps_node/src/control.rs index 3029513b..0fca4061 100644 --- a/lnvps_node/src/control.rs +++ b/lnvps_node/src/control.rs @@ -57,6 +57,10 @@ pub struct ControlState { /// is compared against this, so a request signed for one node cannot be /// replayed against another by setting `Host` to the first node's address. pub base_url: String, + /// The bridge guests are placed on, as LNVPS named it. Kept so the status + /// report observes the bridge this node was actually told to build rather + /// than one the daemon assumes. + pub bridge: String, } impl ControlState { @@ -66,6 +70,15 @@ impl ControlState { control_pubkey, replay: Mutex::new(ReplayGuard::new(REPLAY_WINDOW, REPLAY_CAPACITY)), base_url: format!("https://{addr}"), + bridge: crate::net::DEFAULT_BRIDGE.to_string(), + } + } + + /// Same, for a node whose data plane names a different bridge. + pub fn with_bridge(control_pubkey: PublicKey, addr: SocketAddr, bridge: &str) -> Self { + Self { + bridge: bridge.to_string(), + ..Self::new(control_pubkey, addr) } } } @@ -77,6 +90,11 @@ pub struct NodeStatus { pub version: &'static str, /// Host inventory, the same view `lnvps-node inventory` prints. pub inventory: crate::inventory::Inventory, + /// The data plane as the machine actually has it — read from the machine, + /// never remembered from what was applied. This is the first thing the + /// health gate checks, so a node that quietly lost its tunnel has to be + /// able to say so. + pub dataplane: crate::net::DataPlaneState, } /// The control router, with authentication layered over everything. @@ -177,10 +195,15 @@ fn unauthorized(reason: &str) -> Response { } /// Report what this node is and what it is running. -async fn get_status() -> Json { +async fn get_status(State(state): State>) -> Json { Json(NodeStatus { version: env!("CARGO_PKG_VERSION"), inventory: crate::inventory::Inventory::collect(), + // Observed on demand rather than cached: a cached answer is a report + // that the tunnel was up once, which is exactly the thing the gate must + // not accept. + dataplane: crate::net::observe(&crate::net::SystemCommands, &state.bridge) + .unwrap_or_default(), }) } @@ -244,6 +267,28 @@ mod tests { assert_eq!(code, StatusCode::OK); } + /// The status report observes the bridge LNVPS named, not one the daemon + /// assumes: a node told to use a different bridge would otherwise report on + /// an interface that does not exist and look broken. + #[tokio::test] + async fn the_status_observes_the_bridge_lnvps_named() { + let keys = Keys::generate(); + let addr: SocketAddr = ADDR.parse().unwrap(); + let state = ControlState::with_bridge(keys.public_key(), addr, "br-other"); + assert_eq!(state.bridge, "br-other"); + assert_eq!( + ControlState::new(keys.public_key(), addr).bridge, + "br-lnvps" + ); + + // And the report is servable on a machine with no data plane at all, + // which is exactly the state a node is in before it is configured. + let url = format!("https://{ADDR}/api/v1/status"); + let auth = auth_header(&keys, &url, "GET", b""); + let code = status_of(Arc::new(state), get("/api/v1/status", Some(&auth))).await; + assert_eq!(code, StatusCode::OK); + } + #[tokio::test] async fn an_unsigned_request_is_refused() { let keys = Keys::generate(); diff --git a/lnvps_node/src/lib.rs b/lnvps_node/src/lib.rs index bb0308ea..e5a71472 100644 --- a/lnvps_node/src/lib.rs +++ b/lnvps_node/src/lib.rs @@ -12,11 +12,18 @@ //! - [`tls`] — the node's TLS identity, whose fingerprint LNVPS pins at //! registration, so the node's *replies* are authenticated too. //! - [`inventory`] — what the node reports about the machine. +//! - [`api`] — outbound calls to LNVPS, the only direction that works before +//! there is a tunnel. +//! - [`net`] — applying the data plane LNVPS asked for, with `ip` and `wg`. +//! - [`wgkey`] — the node's WireGuard key, generated here and never sent. //! - [`config`] — configuration, including where the control API may listen. +pub mod api; pub mod config; pub mod control; pub mod control_auth; pub mod credential; pub mod inventory; +pub mod net; pub mod tls; +pub mod wgkey; diff --git a/lnvps_node/src/main.rs b/lnvps_node/src/main.rs index 9a180738..8bbd9749 100644 --- a/lnvps_node/src/main.rs +++ b/lnvps_node/src/main.rs @@ -3,6 +3,7 @@ use std::net::SocketAddr; use std::path::{Path, PathBuf}; use std::sync::Arc; +use std::time::Duration; use anyhow::{Context, Result}; use clap::{Parser, Subcommand}; @@ -31,6 +32,11 @@ enum Command { Inventory, /// Check the configuration and credential without contacting LNVPS. Check, + /// Show or apply the data plane LNVPS has asked this node to run. + Dataplane { + #[command(subcommand)] + action: DataplaneAction, + }, /// Print the node's TLS fingerprint, the value LNVPS pins at registration. Fingerprint { /// State directory holding the identity (defaults to the configured one). @@ -39,6 +45,16 @@ enum Command { }, } +#[derive(Subcommand)] +enum DataplaneAction { + /// Fetch what LNVPS wants and print it, changing nothing. + Show, + /// Fetch it and apply it. + Apply, + /// Print what this machine currently has, read from the machine itself. + Observe, +} + #[tokio::main] async fn main() -> Result<()> { env_logger::init(); @@ -73,6 +89,7 @@ async fn main() -> Result<()> { None => println!("control: not configured (node not yet paired)"), } } + Command::Dataplane { action } => dataplane(&cli.config, action).await?, Command::Fingerprint { state_dir } => { let state_dir = match state_dir { Some(dir) => dir, @@ -91,6 +108,56 @@ async fn main() -> Result<()> { Ok(()) } +/// Show, apply or observe the data plane. +/// +/// Exposed as a command of its own so an operator can see exactly what LNVPS +/// asked for, and re-drive it, without running the daemon or reading its logs. +async fn dataplane(config_path: &Path, action: DataplaneAction) -> Result<()> { + let config = NodeConfig::load(config_path)?; + + // Deliberately before the credential is loaded: "what does this machine + // have?" is the question an operator asks when something is wrong, and it + // must not fail because the token is missing or LNVPS is unreachable. + if let DataplaneAction::Observe = action { + let state = lnvps_node::net::observe( + &lnvps_node::net::SystemCommands, + lnvps_node::net::DEFAULT_BRIDGE, + )?; + println!("{}", serde_json::to_string_pretty(&state)?); + return Ok(()); + } + + let credential = Credential::load_checked(&config.credential)?; + let api = lnvps_node::api::LnvpsApi::new(&config.api_url, &credential)?; + + // Presented before fetching: the document describes a tunnel that does not + // exist until LNVPS has this node's public key, so asking for it first + // would report "no tunnel allocated" on every first run. + let key = lnvps_node::wgkey::load_or_generate(&config.state_dir)?; + if key.generated { + log::info!("Generated this node's tunnel key; presenting it to LNVPS"); + } + api.request_tunnel(&key.public_bytes()).await?; + let desired = api.dataplane().await?; + + match action { + DataplaneAction::Show => println!("{}", serde_json::to_string_pretty(&desired)?), + DataplaneAction::Apply => { + let applied = lnvps_node::net::apply( + &lnvps_node::net::SystemCommands, + &desired, + &key, + &config.state_dir, + )?; + for line in applied { + println!("{line}"); + } + } + DataplaneAction::Observe => unreachable!("handled above, before loading a credential"), + } + Ok(()) +} + /// Start the control API. /// /// The order here is deliberate: everything that can be known to be wrong is @@ -108,6 +175,24 @@ async fn run(config_path: &Path) -> Result<()> { // than starting a listener that can never authorise anything (decision 12). let control_pubkey = control_auth::control_pubkey()?; + // The data plane is applied *before* the listen address is checked, because + // the address being checked for is one this brings into existence: the + // control API binds the tunnel interface, and on a fresh machine the tunnel + // does not exist until now. + let bridge = match apply_dataplane(&config).await { + Ok(bridge) => bridge, + // Not fatal. A node whose tunnel is already up from a previous run must + // keep serving through an LNVPS outage — refusing to start would turn + // an API blip into every node on the platform going dark. + Err(e) => { + log::warn!( + "Could not apply the data plane ({e}); continuing with whatever this machine \ + already has configured" + ); + lnvps_node::net::DEFAULT_BRIDGE.to_string() + } + }; + // Decision 13: the address must belong to the tunnel interface, checked // against the interface itself. let addrs = config::interface_addresses(&control.tunnel_interface)?; @@ -124,5 +209,52 @@ async fn run(config_path: &Path) -> Result<()> { } let addr = SocketAddr::new(control.listen, control.port); - control::serve(Arc::new(ControlState::new(control_pubkey, addr)), addr, tls).await + + // Re-applied on a timer for the same reason the route server reconciles its + // end: guests come and go, and a node that only configured itself at + // startup would route a departed customer's address until it was restarted. + let refresh = config.clone(); + tokio::spawn(async move { + let interval = Duration::from_secs(refresh.heartbeat_secs.max(10)); + loop { + tokio::time::sleep(interval).await; + if let Err(e) = apply_dataplane(&refresh).await { + log::warn!("Data plane refresh failed: {e}"); + } + } + }); + + control::serve( + Arc::new(ControlState::with_bridge(control_pubkey, addr, &bridge)), + addr, + tls, + ) + .await +} + +/// Fetch the data plane and apply it, returning the bridge LNVPS named. +async fn apply_dataplane(config: &NodeConfig) -> Result { + let credential = Credential::load_checked(&config.credential)?; + let api = lnvps_node::api::LnvpsApi::new(&config.api_url, &credential)?; + + let key = lnvps_node::wgkey::load_or_generate(&config.state_dir)?; + if key.generated { + log::warn!( + "Generated a new tunnel key. LNVPS re-pins a node that presents one, but the \ + tunnel stays down until it has been presented." + ); + } + api.request_tunnel(&key.public_bytes()).await?; + let desired = api.dataplane().await?; + + let applied = lnvps_node::net::apply( + &lnvps_node::net::SystemCommands, + &desired, + &key, + &config.state_dir, + )?; + if !applied.is_empty() { + log::debug!("Applied data plane: {}", applied.join("; ")); + } + Ok(desired.bridge) } diff --git a/lnvps_node/src/net.rs b/lnvps_node/src/net.rs new file mode 100644 index 00000000..0b90a37b --- /dev/null +++ b/lnvps_node/src/net.rs @@ -0,0 +1,843 @@ +//! Applying the data plane LNVPS asked for. +//! +//! The daemon configures the machine itself, with `ip` and `wg`, rather than +//! writing files for something else to read. A marketplace node runs on +//! hardware LNVPS does not own: a data plane that depends on the operator +//! having wired it up correctly is one whose mistakes surface as a customer's +//! VM having no network. Applying it here means it re-converges on every +//! refresh instead. +//! +//! Everything is idempotent and stated declaratively — `ip addr replace`, `ip +//! route replace`, `wg set` — so a node that is already correct is not +//! disturbed, and a node that has drifted is corrected without being torn down. +//! +//! Commands go through [`CommandRunner`] because they run as root on somebody +//! else's machine: the exact command issued is the thing worth asserting, which +//! needs the process boundary replaced rather than mocked around. + +use std::collections::HashSet; +use std::path::Path; +use std::process::Command; + +use anyhow::{Context, Result, bail}; +use serde::{Deserialize, Serialize}; + +use crate::wgkey::{self, NodeKey}; + +/// The tunnel interface the node terminates its data plane on. +pub const TUNNEL_INTERFACE: &str = "wg0"; + +/// The bridge guests sit on when LNVPS has not said otherwise — which is only +/// before the first data plane has been fetched. +pub const DEFAULT_BRIDGE: &str = "br-lnvps"; + +/// Runs a command and reports what happened. +pub trait CommandRunner: Send + Sync { + /// Run `program` with `args`, returning `(exit ok, stdout)`. + /// + /// Failure to *launch* is an error; a non-zero exit is not, because several + /// callers here use a failing command as a question ("does this interface + /// exist?") rather than as a fault. + fn run(&self, program: &str, args: &[&str]) -> Result<(bool, String)>; +} + +/// Runs commands as real processes. +pub struct SystemCommands; + +impl CommandRunner for SystemCommands { + fn run(&self, program: &str, args: &[&str]) -> Result<(bool, String)> { + let out = Command::new(program) + .args(args) + .output() + .with_context(|| format!("Cannot run {program}: is it installed?"))?; + let text = if out.status.success() { + String::from_utf8_lossy(&out.stdout).to_string() + } else { + // The failure text, not the empty stdout: a caller reporting why + // something did not apply needs what the tool said. + String::from_utf8_lossy(&out.stderr).to_string() + }; + Ok((out.status.success(), text)) + } +} + +/// The desired data plane, as LNVPS states it. +/// +/// Mirrors `GET /api/v1/node/dataplane`. Fetched and applied as one document +/// because it only makes sense as one: a bridge with no tunnel carries nothing, +/// and a tunnel with no guest routes carries nothing back. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct DesiredDataPlane { + pub tunnel: DesiredTunnel, + pub bridge: String, + /// Gateway addresses this node answers for on the bridge. + #[serde(default)] + pub gateways: Vec, + #[serde(default)] + pub guests: Vec, +} + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct DesiredTunnel { + pub address4: Option, + pub address6: Option, + pub gateway4: Option, + pub gateway6: Option, + /// The route server's key, hex. + pub server_public_key: String, + pub endpoint: String, + pub keepalive: Option, + pub mtu: u16, +} + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct DesiredGuest { + pub address: String, + pub gateway: String, + pub mac: Option, +} + +/// What the machine currently looks like. +/// +/// Reported to LNVPS over the control API, where it is the first thing the +/// health gate checks. Every field is read from the machine, never remembered +/// from what was applied — the point of observing is to catch the case where +/// the two disagree. +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] +pub struct DataPlaneState { + /// Whether `wg0` exists and is up. + pub tunnel_up: bool, + /// Seconds since the last handshake with the route server. `None` when + /// there has never been one, which is the difference between "configured" + /// and "working". + pub last_handshake_secs: Option, + pub tunnel_mtu: Option, + pub bridge_up: bool, + pub forwarding4: bool, + pub forwarding6: bool, + /// Guest addresses actually routed to the bridge. + pub routed_guests: usize, +} + +impl DataPlaneState { + /// Whether this node can carry a customer. + /// + /// A handshake is required, not just an interface: `wg0` comes up happily + /// with a peer that never answers, and a node in that state looks + /// configured while being unreachable. + pub fn healthy(&self) -> bool { + self.tunnel_up && self.last_handshake_secs.is_some() && self.bridge_up && self.forwarding4 + } +} + +/// Apply `desired` to this machine. +/// +/// Returns the commands that were actually run, so `dataplane apply` can show +/// an operator what changed and a test can assert it. +pub fn apply( + runner: &dyn CommandRunner, + desired: &DesiredDataPlane, + key: &NodeKey, + state_dir: &Path, +) -> Result> { + let mut applied = Vec::new(); + apply_tunnel(runner, desired, key, state_dir, &mut applied)?; + apply_bridge(runner, desired, &mut applied)?; + apply_forwarding(runner, &mut applied)?; + Ok(applied) +} + +/// Bring up `wg0` and point the default route down it. +fn apply_tunnel( + runner: &dyn CommandRunner, + desired: &DesiredDataPlane, + key: &NodeKey, + state_dir: &Path, + applied: &mut Vec, +) -> Result<()> { + let mtu = desired.tunnel.mtu.to_string(); + if !link_exists(runner, TUNNEL_INTERFACE)? { + run( + runner, + "ip", + &["link", "add", TUNNEL_INTERFACE, "type", "wireguard"], + applied, + )?; + } + + // The private key is handed over as a path. An argument would be visible in + // `ps` to every user on the machine, and a marketplace node usually has + // more than one login. + let key_file = wgkey::write_private_key_file(state_dir, key)?; + let key_file = key_file.to_string_lossy().to_string(); + run( + runner, + "wg", + &["set", TUNNEL_INTERFACE, "private-key", &key_file], + applied, + )?; + + let server_key = wgkey::parse_public_key(&desired.tunnel.server_public_key)?; + let keepalive = desired.tunnel.keepalive.unwrap_or(0).to_string(); + let mut peer: Vec<&str> = vec![ + "set", + TUNNEL_INTERFACE, + "peer", + &server_key, + "endpoint", + &desired.tunnel.endpoint, + // Everything goes up the tunnel: the node's guests use LNVPS addresses, + // so there is no traffic of theirs that belongs anywhere else. + "allowed-ips", + "0.0.0.0/0,::/0", + ]; + if desired.tunnel.keepalive.is_some() { + peer.extend_from_slice(&["persistent-keepalive", &keepalive]); + } + run(runner, "wg", &peer, applied)?; + + // A peer that is not the route server has no business on this interface. + // It would most likely be a stale key from a re-key, still able to send + // traffic that the node treats as coming from LNVPS. + for stale in stale_peers(runner, &server_key)? { + run( + runner, + "wg", + &["set", TUNNEL_INTERFACE, "peer", &stale, "remove"], + applied, + )?; + } + + for address in [&desired.tunnel.address4, &desired.tunnel.address6] + .into_iter() + .flatten() + { + run( + runner, + "ip", + &["addr", "replace", address, "dev", TUNNEL_INTERFACE], + applied, + )?; + } + + // Not 1500: WireGuard's overhead comes off it, and guessing wrong hangs + // large transfers rather than failing outright. + run( + runner, + "ip", + &["link", "set", TUNNEL_INTERFACE, "mtu", &mtu, "up"], + applied, + )?; + + // No `via`: the tunnel is point-to-point, so the interface names the next + // hop by itself, and naming a gateway would be a second copy of the route + // server's address free to disagree with the one on the peer. + if desired.tunnel.address4.is_some() { + run( + runner, + "ip", + &["route", "replace", "default", "dev", TUNNEL_INTERFACE], + applied, + )?; + } + if desired.tunnel.address6.is_some() { + run( + runner, + "ip", + &["-6", "route", "replace", "default", "dev", TUNNEL_INTERFACE], + applied, + )?; + } + Ok(()) +} + +/// Bring up the guest bridge and route each guest to it. +fn apply_bridge( + runner: &dyn CommandRunner, + desired: &DesiredDataPlane, + applied: &mut Vec, +) -> Result<()> { + let bridge = desired.bridge.as_str(); + if bridge.is_empty() { + bail!("LNVPS did not name a bridge, so there is nothing to put guests on"); + } + if !link_exists(runner, bridge)? { + run( + runner, + "ip", + &["link", "add", bridge, "type", "bridge"], + applied, + )?; + } + // The bridge carries the same payload as the tunnel, so it takes the same + // MTU: a guest that sends 1500 bytes into a 1420-byte tunnel produces a + // connection that opens and then hangs on the first large transfer. + let mtu = desired.tunnel.mtu.to_string(); + run( + runner, + "ip", + &["link", "set", bridge, "mtu", &mtu, "up"], + applied, + )?; + + // The gateway belongs to the range, not to this node, and the guest + // believes it is on-link. Held as a host address so the node answers for it + // without claiming the rest of the range is local — the other addresses in + // it live on other nodes, up the tunnel. + for gateway in &desired.gateways { + let addr = host_prefix(gateway)?; + run( + runner, + "ip", + &["addr", "replace", &addr, "dev", bridge], + applied, + )?; + } + + // The guest thinks its neighbours are on-link and will ARP for them; proxy + // ARP is what lets the node answer and pull that traffic up the tunnel + // instead of it disappearing into a link that has no such address. + for knob in [ + "net.ipv4.conf.NAME.proxy_arp=1", + "net.ipv6.conf.NAME.proxy_ndp=1", + ] { + // Interface names appear in sysctl keys with dots replaced, or the key + // itself becomes ambiguous. + let setting = knob.replace("NAME", &bridge.replace('.', "/")); + run(runner, "sysctl", &["-w", &setting], applied)?; + } + + // What belongs on this bridge: the guests, plus the gateways the node + // answers for. Both are kept so the stale sweep below cannot delete the + // bridge's own addressing while tidying up after a departed guest. + let mut want: HashSet = desired.guests.iter().map(|g| g.address.clone()).collect(); + for gateway in &desired.gateways { + want.insert(host_prefix(gateway)?); + } + let want: HashSet = want; + // Sorted, so a node applying the same document twice runs the same + // commands in the same order and a diff of two runs means something. + let mut guests: Vec<&String> = desired.guests.iter().map(|g| &g.address).collect(); + guests.sort(); + for address in guests { + run( + runner, + "ip", + &["route", "replace", address, "dev", bridge], + applied, + )?; + } + // A guest that has been deleted or moved must stop being routed here at + // once: its address goes back in the pool and may already be somebody + // else's. + for stale in stale_routes(runner, bridge, &want)? { + run( + runner, + "ip", + &["route", "del", &stale, "dev", bridge], + applied, + )?; + } + Ok(()) +} + +/// A node that does not forward is a node whose guests have no network at all. +fn apply_forwarding(runner: &dyn CommandRunner, applied: &mut Vec) -> Result<()> { + for setting in ["net.ipv4.ip_forward=1", "net.ipv6.conf.all.forwarding=1"] { + run(runner, "sysctl", &["-w", setting], applied)?; + } + Ok(()) +} + +/// Read back what the machine actually has. +pub fn observe(runner: &dyn CommandRunner, bridge: &str) -> Result { + let (tunnel_up, tunnel_mtu) = link_state(runner, TUNNEL_INTERFACE)?; + let (bridge_up, _) = link_state(runner, bridge)?; + Ok(DataPlaneState { + tunnel_up, + tunnel_mtu, + last_handshake_secs: last_handshake(runner)?, + bridge_up, + forwarding4: sysctl_enabled(runner, "net.ipv4.ip_forward")?, + forwarding6: sysctl_enabled(runner, "net.ipv6.conf.all.forwarding")?, + routed_guests: routed_addresses(runner, bridge)?.len(), + }) +} + +/// Whether a link exists at all. +fn link_exists(runner: &dyn CommandRunner, name: &str) -> Result { + Ok(runner.run("ip", &["link", "show", name])?.0) +} + +/// Whether a link is up, and its MTU. +fn link_state(runner: &dyn CommandRunner, name: &str) -> Result<(bool, Option)> { + let (ok, out) = runner.run("ip", &["-j", "link", "show", name])?; + if !ok { + return Ok((false, None)); + } + let links: Vec = serde_json::from_str(&out).unwrap_or_default(); + let Some(link) = links.first() else { + return Ok((false, None)); + }; + // `operstate` rather than the UP flag: an interface can be administratively + // up with no carrier, and for a tunnel that is exactly the broken case. + let up = link + .get("flags") + .and_then(|f| f.as_array()) + .map(|f| f.iter().any(|v| v.as_str() == Some("UP"))) + .unwrap_or(false); + let mtu = link.get("mtu").and_then(|m| m.as_u64()).map(|m| m as u32); + Ok((up, mtu)) +} + +/// Seconds since the route server last completed a handshake. +fn last_handshake(runner: &dyn CommandRunner) -> Result> { + let (ok, out) = runner.run("wg", &["show", TUNNEL_INTERFACE, "latest-handshakes"])?; + if !ok { + return Ok(None); + } + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + let latest = out + .lines() + .filter_map(|line| line.split_whitespace().nth(1)) + .filter_map(|t| t.parse::().ok()) + // Zero means "never", not "in 1970"; reporting an age of half a century + // would look like a stale tunnel rather than one that has never worked. + .filter(|t| *t > 0) + .max(); + Ok(latest.map(|t| now.saturating_sub(t))) +} + +/// Whether a sysctl is on. +fn sysctl_enabled(runner: &dyn CommandRunner, name: &str) -> Result { + let (ok, out) = runner.run("sysctl", &["-n", name])?; + Ok(ok && out.trim() == "1") +} + +/// Peers configured on the tunnel that are not the route server. +fn stale_peers(runner: &dyn CommandRunner, server_key: &str) -> Result> { + let (ok, out) = runner.run("wg", &["show", TUNNEL_INTERFACE, "peers"])?; + if !ok { + return Ok(vec![]); + } + Ok(out + .lines() + .map(str::trim) + .filter(|l| !l.is_empty() && *l != server_key) + .map(str::to_string) + .collect()) +} + +/// Addresses currently routed to the bridge. +fn routed_addresses(runner: &dyn CommandRunner, bridge: &str) -> Result> { + let mut out = Vec::new(); + // Both families asked for separately: `ip route show` is IPv4 only, so a v6 + // guest would look unrouted on every pass and be re-added forever. + for family in ["-4", "-6"] { + let (ok, text) = runner.run("ip", &[family, "-j", "route", "show", "dev", bridge])?; + if !ok { + continue; + } + let routes: Vec = serde_json::from_str(&text).unwrap_or_default(); + for route in routes { + let Some(dst) = route.get("dst").and_then(|d| d.as_str()) else { + continue; + }; + if dst == "default" { + continue; + } + out.push(if dst.contains('/') { + dst.to_string() + } else { + host_prefix(dst)? + }); + } + } + Ok(out) +} + +/// Routes on the bridge that no guest accounts for. +fn stale_routes( + runner: &dyn CommandRunner, + bridge: &str, + want: &HashSet, +) -> Result> { + Ok(routed_addresses(runner, bridge)? + .into_iter() + .filter(|r| !want.contains(r)) + .collect()) +} + +/// `203.0.113.1` -> `203.0.113.1/32`, and the v6 equivalent. +fn host_prefix(address: &str) -> Result { + if address.contains('/') { + return Ok(address.to_string()); + } + let ip: std::net::IpAddr = address + .parse() + .with_context(|| format!("{address} is not an IP address"))?; + Ok(match ip { + std::net::IpAddr::V4(v4) => format!("{v4}/32"), + std::net::IpAddr::V6(v6) => format!("{v6}/128"), + }) +} + +/// Run a command that must succeed, recording it. +fn run( + runner: &dyn CommandRunner, + program: &str, + args: &[&str], + applied: &mut Vec, +) -> Result<()> { + let (ok, out) = runner.run(program, args)?; + let line = format!("{program} {}", args.join(" ")); + if !ok { + bail!("{line}: {}", out.trim()); + } + applied.push(line); + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Mutex; + + /// A machine that answers `ip`/`wg` queries from a fixed script and records + /// everything it was asked to change. + /// + /// The commands run as root on somebody else's hardware, so what is worth + /// asserting is the exact command issued — which needs the process boundary + /// replaced, not mocked around. + struct FakeMachine { + log: Mutex>, + /// Substring -> canned (ok, stdout). + answers: Vec<(&'static str, bool, &'static str)>, + } + + impl FakeMachine { + fn new(answers: Vec<(&'static str, bool, &'static str)>) -> Self { + Self { + log: Mutex::new(Vec::new()), + answers, + } + } + + fn ran(&self) -> Vec { + self.log.lock().unwrap().clone() + } + } + + impl CommandRunner for FakeMachine { + fn run(&self, program: &str, args: &[&str]) -> Result<(bool, String)> { + let line = format!("{program} {}", args.join(" ")); + self.log.lock().unwrap().push(line.clone()); + for (needle, ok, out) in &self.answers { + if line.contains(needle) { + return Ok((*ok, out.to_string())); + } + } + Ok((true, String::new())) + } + } + + fn desired() -> DesiredDataPlane { + DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some("10.66.0.2/32".to_string()), + address6: Some("fd00:66::2/128".to_string()), + gateway4: Some("10.66.0.1".to_string()), + gateway6: Some("fd00:66::1".to_string()), + server_public_key: hex::encode([0xab; 32]), + endpoint: "rs1.example:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + bridge: "br-lnvps".to_string(), + gateways: vec!["203.0.113.1".to_string()], + guests: vec![DesiredGuest { + address: "203.0.113.5/32".to_string(), + gateway: "203.0.113.1".to_string(), + mac: Some("aa:bb:cc:dd:ee:ff".to_string()), + }], + } + } + + fn key(dir: &Path) -> NodeKey { + wgkey::load_or_generate(dir).unwrap() + } + + /// A node with nothing configured must end up with the whole data plane: + /// tunnel, addresses, MTU, default route, bridge, gateway, guest route and + /// forwarding. Any one of them missing is a customer with no network. + #[test] + fn a_bare_machine_gets_the_whole_data_plane() { + let dir = tempfile::tempdir().unwrap(); + // Neither interface exists yet. + let machine = FakeMachine::new(vec![("ip link show", false, "does not exist")]); + let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); + let script = applied.join("\n"); + + assert!( + script.contains("ip link add wg0 type wireguard"), + "{script}" + ); + assert!(script.contains("wg set wg0 private-key"), "{script}"); + // Everything goes up the tunnel: the guests use LNVPS addresses, so no + // traffic of theirs belongs anywhere else. + assert!(script.contains("allowed-ips 0.0.0.0/0,::/0"), "{script}"); + assert!(script.contains("persistent-keepalive 25"), "{script}"); + assert!( + script.contains("ip addr replace 10.66.0.2/32 dev wg0"), + "{script}" + ); + assert!( + script.contains("ip addr replace fd00:66::2/128 dev wg0"), + "{script}" + ); + assert!(script.contains("ip link set wg0 mtu 1420 up"), "{script}"); + assert!( + script.contains("ip route replace default dev wg0"), + "{script}" + ); + assert!( + script.contains("ip -6 route replace default dev wg0"), + "{script}" + ); + + assert!( + script.contains("ip link add br-lnvps type bridge"), + "{script}" + ); + // The bridge takes the tunnel's MTU: a guest sending 1500 bytes into a + // 1420-byte tunnel opens a connection and then hangs on a large one. + assert!( + script.contains("ip link set br-lnvps mtu 1420 up"), + "{script}" + ); + // The gateway belongs to the range, not the node, and is held as a host + // address so the node answers for it without claiming the rest of the + // range is local. + assert!( + script.contains("ip addr replace 203.0.113.1/32 dev br-lnvps"), + "{script}" + ); + assert!(script.contains("proxy_arp=1"), "{script}"); + assert!(script.contains("proxy_ndp=1"), "{script}"); + assert!( + script.contains("ip route replace 203.0.113.5/32 dev br-lnvps"), + "{script}" + ); + assert!(script.contains("net.ipv4.ip_forward=1"), "{script}"); + assert!( + script.contains("net.ipv6.conf.all.forwarding=1"), + "{script}" + ); + } + + /// The private key reaches `wg` as a path, never as an argument: arguments + /// are visible in `ps` to every user on the machine. + #[test] + fn the_private_key_is_never_an_argument() { + let dir = tempfile::tempdir().unwrap(); + let node_key = key(dir.path()); + let machine = FakeMachine::new(vec![]); + apply(&machine, &desired(), &node_key, dir.path()).unwrap(); + + let script = machine.ran().join("\n"); + assert!( + !script.contains(&node_key.private_base64()), + "the private key was passed on a command line" + ); + assert!(script.contains("private-key"), "{script}"); + } + + /// An interface that already exists is configured, not recreated: + /// recreating it drops the tunnel every time the node refreshes. + #[test] + fn an_existing_interface_is_not_recreated() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![("ip link show", true, "")]); + let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); + let script = applied.join("\n"); + assert!(!script.contains("ip link add"), "{script}"); + assert!(script.contains("ip addr replace"), "{script}"); + } + + /// A peer that is not the route server has no business on this interface — + /// most likely a stale key from a re-key, still able to send traffic the + /// node would treat as LNVPS's. + #[test] + fn a_stale_peer_is_removed() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![("wg show wg0 peers", true, "c3RyYXk=\n")]); + let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); + let script = applied.join("\n"); + assert!( + script.contains("wg set wg0 peer c3RyYXk= remove"), + "{script}" + ); + } + + /// A guest that has been deleted or moved must stop being routed here at + /// once: its address goes back in the pool and may already be somebody + /// else's. The bridge's own gateway must survive that sweep. + #[test] + fn a_departed_guest_stops_being_routed_but_the_gateway_stays() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![( + "-4 -j route show dev br-lnvps", + true, + r#"[{"dst":"203.0.113.5"},{"dst":"203.0.113.9"},{"dst":"203.0.113.1"}]"#, + )]); + let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); + let script = applied.join("\n"); + assert!( + script.contains("ip route del 203.0.113.9/32 dev br-lnvps"), + "{script}" + ); + assert!( + !script.contains("ip route del 203.0.113.5/32"), + "a guest that is still here was unrouted" + ); + assert!( + !script.contains("ip route del 203.0.113.1/32"), + "the bridge's own gateway was deleted" + ); + } + + /// A command that fails stops the run and says which one: half a data plane + /// applied silently is worse than none, because it looks configured. + #[test] + fn a_failing_command_is_reported() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![ + ("ip link show", false, ""), + ("ip link add wg0", false, "RTNETLINK answers: not permitted"), + ]); + let err = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap_err(); + assert!(format!("{err}").contains("ip link add wg0"), "{err}"); + assert!(format!("{err}").contains("not permitted"), "{err}"); + } + + /// A document naming no bridge would silently configure a tunnel with + /// nothing behind it. + #[test] + fn a_document_without_a_bridge_is_refused() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![]); + let plane = DesiredDataPlane { + bridge: String::new(), + ..desired() + }; + assert!(apply(&machine, &plane, &key(dir.path()), dir.path()).is_err()); + } + + /// A single-stack pool must not produce a default route for the family it + /// has no address in — that route would black-hole traffic instead of + /// letting the machine's own routing handle it. + #[test] + fn a_single_stack_tunnel_only_routes_its_own_family() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![]); + let mut plane = desired(); + plane.tunnel.address6 = None; + let applied = apply(&machine, &plane, &key(dir.path()), dir.path()).unwrap(); + let script = applied.join("\n"); + assert!(script.contains("ip route replace default dev wg0")); + assert!(!script.contains("ip -6 route replace default"), "{script}"); + } + + /// A gateway that is not an address is reported against the value, not as + /// a failing `ip` command: LNVPS sent it, and the node has to say which + /// part of the document it could not use. + #[test] + fn a_gateway_that_is_not_an_address_is_reported() { + let dir = tempfile::tempdir().unwrap(); + let machine = FakeMachine::new(vec![]); + let plane = DesiredDataPlane { + gateways: vec!["not-an-address".to_string()], + ..desired() + }; + let err = apply(&machine, &plane, &key(dir.path()), dir.path()).unwrap_err(); + assert!(format!("{err:#}").contains("not-an-address"), "{err:#}"); + } + + /// Observation reads the machine rather than remembering what was applied: + /// the point of observing is to catch the case where the two disagree. + #[test] + fn observation_reports_what_the_machine_has() { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let machine = FakeMachine::new(vec![ + ( + "-j link show wg0", + true, + r#"[{"ifname":"wg0","flags":["POINTOPOINT","NOARP","UP","LOWER_UP"],"mtu":1420}]"#, + ), + ( + "-j link show br-lnvps", + true, + r#"[{"ifname":"br-lnvps","flags":["BROADCAST","MULTICAST","UP"],"mtu":1420}]"#, + ), + ( + "wg show wg0 latest-handshakes", + true, + Box::leak(format!("peerkey\t{}\n", now - 12).into_boxed_str()), + ), + ("sysctl -n", true, "1\n"), + ( + "-4 -j route show dev br-lnvps", + true, + r#"[{"dst":"203.0.113.5"}]"#, + ), + ]); + + let state = observe(&machine, "br-lnvps").unwrap(); + assert!(state.tunnel_up); + assert_eq!(state.tunnel_mtu, Some(1420)); + assert!(state.last_handshake_secs.unwrap() <= 13); + assert!(state.bridge_up); + assert!(state.forwarding4 && state.forwarding6); + assert_eq!(state.routed_guests, 1); + assert!(state.healthy()); + } + + /// `wg0` comes up happily with a peer that never answers, so an interface + /// that has never handshaken is configured, not working — and a node in + /// that state must not be called healthy. + #[test] + fn a_tunnel_that_has_never_handshaken_is_not_healthy() { + let machine = FakeMachine::new(vec![ + ( + "-j link show", + true, + r#"[{"ifname":"wg0","flags":["UP"],"mtu":1420}]"#, + ), + // Zero means never, not 1970: reporting an age of half a century + // would look like a stale tunnel rather than one never used. + ("latest-handshakes", true, "peerkey\t0\n"), + ("sysctl -n", true, "1\n"), + ]); + let state = observe(&machine, "br-lnvps").unwrap(); + assert!(state.tunnel_up); + assert_eq!(state.last_handshake_secs, None); + assert!(!state.healthy()); + } + + /// A machine with nothing configured reports nothing configured, rather + /// than failing: "not set up yet" is a state the gate has to be able to + /// read. + #[test] + fn an_unconfigured_machine_observes_cleanly() { + let machine = FakeMachine::new(vec![("", false, "does not exist")]); + let state = observe(&machine, "br-lnvps").unwrap(); + assert_eq!(state, DataPlaneState::default()); + assert!(!state.healthy()); + } +} diff --git a/lnvps_node/src/wgkey.rs b/lnvps_node/src/wgkey.rs new file mode 100644 index 00000000..05c8c120 --- /dev/null +++ b/lnvps_node/src/wgkey.rs @@ -0,0 +1,292 @@ +//! The node's WireGuard keypair. +//! +//! Generated here and kept here: LNVPS is told the public half and never sees +//! the private one. That is the whole reason a node presents a key rather than +//! being issued one — an operator's machine that LNVPS could impersonate would +//! make the tunnel's authentication decorative. +//! +//! The key is generated in-process rather than by shelling out to `wg genkey`, +//! so a node with a broken or missing `wg` fails when it tries to *configure* +//! the interface, with that error, instead of failing here with a confusing +//! one — and so the tests do not need a fake for key generation. +//! +//! These few lines duplicate `lnvps_api_common::wireguard` rather than +//! depending on it: that crate pulls in the database, axum and the payment +//! stack, none of which belong on somebody else's hardware. + +use std::fs; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, bail}; +use base64::Engine; +use base64::engine::general_purpose::STANDARD; +use rand::TryRngCore; +use x25519_dalek::{PublicKey, StaticSecret}; + +/// A node's WireGuard identity. +pub struct NodeKey { + secret: StaticSecret, + /// True when this run created the key, so the caller can say so once rather + /// than logging it on every start. + pub generated: bool, +} + +impl std::fmt::Debug for NodeKey { + /// Hand-written: the derived form would print the private key, and this + /// type travels through anyhow errors and log lines. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("NodeKey()") + } +} + +impl NodeKey { + /// The public half, base64, as WireGuard writes keys. + pub fn public_base64(&self) -> String { + STANDARD.encode(PublicKey::from(&self.secret).as_bytes()) + } + + /// The public half as raw bytes, which is how LNVPS stores it. + pub fn public_bytes(&self) -> [u8; 32] { + *PublicKey::from(&self.secret).as_bytes() + } + + /// The private half, base64. + /// + /// Only for handing to `wg` through a file; it is deliberately not + /// reachable from `Debug` or `Display`. + pub fn private_base64(&self) -> String { + STANDARD.encode(self.secret.to_bytes()) + } +} + +/// Where the key lives inside the state directory. +pub fn key_path(state_dir: &Path) -> PathBuf { + state_dir.join("tunnel.key") +} + +/// Load the node's key, generating one on first use. +/// +/// A regenerated key is not fatal: LNVPS re-pins a node that presents a new one +/// rather than refusing it, because a machine restored from backup that can +/// never be reached again is worse than a re-pin. The caller still warns, since +/// the tunnel stays down until the new key has been presented. +pub fn load_or_generate(state_dir: &Path) -> Result { + let path = key_path(state_dir); + if path.exists() { + crate::credential::check_permissions(&path)?; + let contents = fs::read_to_string(&path) + .with_context(|| format!("Cannot read tunnel key {}", path.display()))?; + return parse(&contents, &path); + } + + fs::create_dir_all(state_dir) + .with_context(|| format!("Cannot create state directory {}", state_dir.display()))?; + let mut bytes = [0u8; 32]; + rand::rngs::OsRng + .try_fill_bytes(&mut bytes) + .map_err(|e| anyhow::anyhow!("No system randomness available for a tunnel key: {e}"))?; + let secret = StaticSecret::from(bytes); + write_secret(&path, &STANDARD.encode(secret.to_bytes()))?; + Ok(NodeKey { + secret, + generated: true, + }) +} + +/// Parse a stored key. +pub fn parse(contents: &str, path: &Path) -> Result { + let raw = STANDARD + .decode(contents.trim()) + .with_context(|| format!("Tunnel key {} is not base64", path.display()))?; + let bytes: [u8; 32] = raw.as_slice().try_into().map_err(|_| { + anyhow::anyhow!( + "Tunnel key {} is {} bytes; a WireGuard key is 32", + path.display(), + raw.len() + ) + })?; + Ok(NodeKey { + secret: StaticSecret::from(bytes), + generated: false, + }) +} + +/// Write `contents` to `path` readable only by its owner. +/// +/// The mode is set **before** the key is written, not after: a key that is +/// world-readable for even the moment between the two is a key that a process +/// watching the directory has already read. +#[cfg(unix)] +fn write_secret(path: &Path, contents: &str) -> Result<()> { + use std::io::Write; + use std::os::unix::fs::OpenOptionsExt; + + let mut file = fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .open(path) + .with_context(|| format!("Cannot write tunnel key {}", path.display()))?; + file.write_all(contents.as_bytes())?; + Ok(()) +} + +/// Non-Unix hosts have no mode bits. Nodes are Linux-only; this exists so the +/// crate still builds on a developer's other machine. +#[cfg(not(unix))] +fn write_secret(path: &Path, contents: &str) -> Result<()> { + fs::write(path, contents).with_context(|| format!("Cannot write tunnel key {}", path.display())) +} + +/// Write the private key to a file `wg` can read, inside the state directory. +/// +/// `wg set` takes the key as a **path**, never as an argument, because an +/// argument is visible in `ps` to every user on the machine — and a marketplace +/// node usually has more than one login. +pub fn write_private_key_file(state_dir: &Path, key: &NodeKey) -> Result { + let path = state_dir.join("tunnel.key.wg"); + write_secret(&path, &key.private_base64())?; + Ok(path) +} + +/// Reject a server key that is not a WireGuard key, before it reaches `wg`. +pub fn parse_public_key(value: &str) -> Result { + // LNVPS sends hex; `wg` speaks base64. Converting here keeps the wire + // format consistent with the rest of the node API, where keys are hex. + let raw = hex::decode(value.trim()) + .with_context(|| format!("Server public key {value} is not hex"))?; + if raw.len() != 32 { + bail!( + "Server public key is {} bytes; a WireGuard key is 32", + raw.len() + ); + } + Ok(STANDARD.encode(raw)) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A key generated here must be the one that comes back, and the public + /// half must be derived from it rather than stored beside it — a stored + /// copy is free to disagree with the key it claims to describe. + #[test] + fn a_key_survives_a_restart() { + let dir = tempfile::tempdir().unwrap(); + let first = load_or_generate(dir.path()).unwrap(); + assert!(first.generated); + + let second = load_or_generate(dir.path()).unwrap(); + assert!(!second.generated, "a second start generated a new key"); + assert_eq!(first.public_base64(), second.public_base64()); + assert_eq!(first.private_base64(), second.private_base64()); + assert_eq!(first.public_bytes(), second.public_bytes()); + } + + /// The key is written owner-only from the moment it exists: a marketplace + /// node usually has more than one login on it. + #[cfg(unix)] + #[test] + fn a_generated_key_is_not_readable_by_other_users() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + load_or_generate(dir.path()).unwrap(); + let mode = fs::metadata(key_path(dir.path())) + .unwrap() + .permissions() + .mode(); + assert_eq!(mode & 0o077, 0, "mode {:04o}", mode & 0o7777); + } + + /// A key file that is readable by everyone is refused rather than used: the + /// tunnel it authenticates is the node's whole security boundary. + #[cfg(unix)] + #[test] + fn a_world_readable_key_is_refused() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + load_or_generate(dir.path()).unwrap(); + fs::set_permissions(key_path(dir.path()), fs::Permissions::from_mode(0o644)).unwrap(); + assert!(load_or_generate(dir.path()).is_err()); + } + + /// The key must not reach a log line or an anyhow error, both of which + /// this type travels through. + #[test] + fn the_key_is_never_printed() { + let dir = tempfile::tempdir().unwrap(); + let key = load_or_generate(dir.path()).unwrap(); + let printed = format!("{key:?}"); + assert!(!printed.contains(&key.private_base64()), "{printed}"); + assert!(printed.contains("redacted"), "{printed}"); + } + + /// A state directory that cannot hold a key, and a key file that cannot be + /// read, are reported against the path rather than as a bare io error — the + /// operator has to know which file to fix. + #[test] + fn an_unusable_state_directory_is_reported_with_its_path() { + let dir = tempfile::tempdir().unwrap(); + + // A file where the state directory should be. + let as_file = dir.path().join("state"); + fs::write(&as_file, "").unwrap(); + let err = load_or_generate(&as_file).unwrap_err(); + assert!(format!("{err:#}").contains("state"), "{err:#}"); + + // A directory where the key file should be: it exists, so it is read, + // and reading it fails. + let state = dir.path().join("other"); + fs::create_dir_all(key_path(&state)).unwrap(); + let err = load_or_generate(&state).unwrap_err(); + assert!(format!("{err:#}").contains("tunnel.key"), "{err:#}"); + } + + /// Anything that is not 32 bytes of base64 is not a WireGuard key, and + /// saying so here beats a handshake that never completes. + #[test] + fn a_malformed_key_is_refused() { + let path = Path::new("/tmp/tunnel.key"); + assert!(parse("not base64!", path).is_err()); + assert!(parse(&STANDARD.encode([1u8; 16]), path).is_err()); + assert!(parse(&STANDARD.encode([1u8; 32]), path).is_ok()); + } + + /// The server's key arrives as hex and reaches `wg` as base64; a value that + /// is neither is refused before it is written into a command. + #[test] + fn a_server_key_is_converted_and_checked() { + assert_eq!( + parse_public_key(&hex::encode([0xab; 32])).unwrap(), + STANDARD.encode([0xab; 32]) + ); + assert!(parse_public_key("nonsense").is_err()); + assert!(parse_public_key(&hex::encode([0xab; 16])).is_err()); + } + + /// `wg set` takes a path, never an argument: an argument is visible in `ps` + /// to every user on the machine. + #[cfg(unix)] + #[test] + fn the_key_file_handed_to_wg_is_owner_only() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + let key = load_or_generate(dir.path()).unwrap(); + let path = write_private_key_file(dir.path(), &key).unwrap(); + assert_eq!(fs::read_to_string(&path).unwrap(), key.private_base64()); + let mode = fs::metadata(&path).unwrap().permissions().mode(); + assert_eq!(mode & 0o077, 0, "mode {:04o}", mode & 0o7777); + + // A key file that cannot be written is named, not swallowed: `wg` would + // otherwise fail later with a path the operator never chose. + let blocked = tempfile::tempdir().unwrap(); + fs::create_dir_all(blocked.path().join("tunnel.key.wg")).unwrap(); + let err = write_private_key_file(blocked.path(), &key).unwrap_err(); + assert!(format!("{err:#}").contains("tunnel.key.wg"), "{err:#}"); + } +} diff --git a/work/marketplace.md b/work/marketplace.md index 83ea27b3..80aad514 100644 --- a/work/marketplace.md +++ b/work/marketplace.md @@ -831,12 +831,91 @@ for want of a real box, as before. from a router is drift to report, not an allocation to forget. - Route the guest prefixes at the route server towards the peer. -#### 4c — Node data plane + health gate (L) ⬅ NEXT -- Node side: `wg0` + `br-lnvps`, default route into the tunnel, anti-spoof and anti-LAN-access - rules, MTU/MSS clamp. -- Health gate: the host is only enabled after an end-to-end reachability probe from the route - server through the tunnel to a test guest IP. Failure leaves the node approved but unusable, - which is the safe direction. +#### 4c — Node data plane + health gate (XL — split into 4c1/4c2/4c3) + +Sized as one increment it is XL. Three decisions taken before starting made it so, each +deliberately: + +- **The daemon applies the configuration itself**, with `ip` and `wg`, rather than writing + `wg-quick` files for something else to read. A marketplace node runs on hardware LNVPS does + not own, so a data plane that depends on the operator having wired it up correctly is one + whose mistakes surface as a customer's VM having no network. The daemon re-converges instead. +- **Both `nft` and `iptables` are supported**, detected at runtime, because a node is somebody + else's machine and refusing the ones that run iptables would refuse real hardware. It costs a + second dialect, which is why the firewall is its own increment. +- **The health gate spawns a real guest and pings it** rather than asking the node how it + thinks it is doing. The node self-reporting "bridge up, forwarding on" cannot catch a bridge + with no path to the tunnel, which is exactly the mistake worth catching before a customer + finds it. + +The node also has no outbound API client yet (that was 2c, still pending), so 4c1 builds the +one call it needs rather than waiting. + +#### 4c1 — Node data plane (L) ✅ DONE + +What the build settled beyond the plan: +- **The data plane is applied before the listener binds.** The control API binds an address of + the tunnel interface, and on a fresh machine that interface does not exist until the daemon + has fetched and applied its document — so the startup order is apply, then check, then serve. + A failure to fetch is a warning rather than a fatal error: a node whose tunnel is already up + from a previous run must keep serving through an LNVPS outage, or an API blip takes every node + on the platform dark at once. +- **The gateway a guest uses belongs to its range, not to the node.** The guest is configured + with the range's gateway and believes it is on-link, so the node holds that address on the + bridge as a **host** address and turns on proxy ARP/NDP. Holding the whole range instead would + make the node believe every other node's guests were local, and their traffic would disappear + into the bridge instead of going up the tunnel. +- **The bridge takes the tunnel's MTU.** A guest sending 1500 bytes into a 1420-byte tunnel gets + a connection that opens and then hangs on the first large transfer — the worst failure shape + there is, because everything looks fine until it does not. +- **A peer that is not the route server is removed from `wg0`**, most likely a stale key left by + a re-key, which would otherwise still be able to send traffic the node treats as LNVPS's. +- **Routes for departed guests are swept**, since a released address goes straight back in the + pool and may already be somebody else's; the bridge's own gateway addresses are excluded from + that sweep so tidying up after a guest cannot take the bridge's addressing with it. +- **Observation reads the machine, never a cache.** `/api/v1/status` runs the queries on + demand: a cached answer reports that the tunnel was up once, which is exactly what the health + gate must not accept. A tunnel that has never handshaken is reported as configured but not + working, because `wg0` comes up perfectly happily with a peer that never answers. +- **The node generates its key in-process** rather than shelling out to `wg genkey`, so a + missing `wg` fails when the interface is configured, with that error. The private key reaches + `wg` as a **path**, never an argument: arguments are visible in `ps` to every user on the + machine, and a marketplace node usually has more than one login. +- **`lnvps-node dataplane observe` deliberately needs no credential**, because "what does this + machine actually have?" is the question asked when something is already broken. + +Testing note: `net.rs` runs commands through a `CommandRunner`, faked in tests to answer `ip` +and `wg` queries from a script and record everything it was asked to change. These commands run +as root on somebody else's hardware, so the exact command issued is the thing worth asserting. + +#### 4c1 — original scope +- `GET /api/v1/node/dataplane` (node token): the desired state in one document — the tunnel + (key, addresses, endpoint, MTU, keepalive), the bridge, and the guest addresses assigned to + this node. One call rather than three: the node applies these together or not at all, and a + document that can be half-fetched is a data plane that can be half-applied. +- Node keypair: generated on first use into the state directory, `0600`, public half presented + to `POST /api/v1/node/tunnel`. The private half never leaves the machine. +- `net.rs`: `wg0`, `br-lnvps`, the default route into the tunnel, MTU, and IP forwarding — + idempotent, applied on startup and on every refresh, through a command runner that tests can + record. These commands run as root on somebody else's machine; the exact command is the thing + worth asserting. +- `lnvps-node dataplane show|apply` so an operator can see and re-drive it without the daemon. +- `/api/v1/status` reports the observed data plane, which 4c3's gate reads as its first check. + +#### 4c2 — Anti-spoof + anti-LAN firewall (M/L) ⬅ NEXT +- One ruleset the daemon owns wholesale, in `nft` where available and `iptables` where not, + with the backend detected once and reported in status. +- Guests may not reach the operator's own LAN or the node's management addresses; may not + source traffic as an address LNVPS did not assign them; MSS clamped to the tunnel's MTU. +- The guest address set comes from 4c1's document, so the boundary is LNVPS's list, not + something the node infers. + +#### 4c3 — Health gate (M/L) +- A probe guest is provisioned through the ordinary path onto the new node, given a real + address, and pinged from the route server through the tunnel. Only then is the host enabled. +- Failure leaves the node approved but unusable, with the failing step named. That is the safe + direction: a node that never carries a customer is a support conversation, a node that + carries one badly is an outage. ### Increment 5 — Confidential computing: attestation + encrypted disks (L) - Verify SEV-SNP attestation reports (`sev` crate) / TDX quotes (`dcap-qvl` crate) against From 65439cb4cdd80c619db0dd309cecc968ed913840 Mon Sep 17 00:00:00 2001 From: v0l Date: Fri, 7 Aug 2026 00:43:00 +0100 Subject: [PATCH 2/3] feat(marketplace): netlink, a namespace of its own, and a tunnel that is proved MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes to the node's data plane, and the test that made them necessary. **Netlink instead of `ip`.** The daemon now talks to the kernel directly: `rtnetlink` for links, addresses and routes, WireGuard's own netlink interface for the tunnel, and `/proc/sys` for the forwarding knobs. `ip` is a program that formats netlink messages and formats the replies back into text for us to parse — going direct drops a dependency on iproute2's presence and version, drops the output parsing that changes between releases, and turns "a line of English on stderr" into kernel error codes. **The data plane lives in its own network namespace.** The first cut configured the *machine's* network: it replaced the operator's default route and turned on forwarding machine-wide, on hardware that is frequently not only an LNVPS node. Now `wg0` and `br-lnvps` live in an `lnvps` namespace: - their default route stays theirs, and the forwarding and proxy-ARP knobs are ours alone; - guests cannot reach the operator's network — not because a rule forbids it, which can be mis-ordered or flushed, but because no interface leads there; - a tunnel that is down means no path at all, instead of customer traffic leaking out the operator's uplink sourced from LNVPS addresses, which looks like spoofing to their upstream and can get *their* connection null-routed. `wg0` is created in the machine's namespace and then moved, because a WireGuard interface keeps its UDP socket where it was created: the encrypted outer traffic still leaves by the operator's uplink while everything carried over the tunnel is isolated. **The bridge name is no longer sent to the node.** Both sides hold it as a constant. The daemon needs the name before it has ever spoken to LNVPS — `dataplane observe` takes no credential — so a document that could name a different bridge would leave the node holding two answers to one question. **And a harness that sends real packets.** `lnvps_e2e/tests/tunnel_netns.rs` builds both ends out of network namespaces: the route server configured through its real code path, the node through its real netlink calls, a guest on the bridge behind it. It pings the node over the tunnel and then the guest through it, which is the path a customer's traffic takes. It earned its place immediately by finding four things no unit test could: - the namespace was pinned from `/proc/self/ns/net`, which in a multi-threaded process is the *process's* namespace — so every "isolated" interface was silently landing in the operator's own network; - WireGuard's netlink calls ran outside the namespace the interface had been moved into, reporting "no such device" about an interface that plainly existed; - giving an interface an address makes the kernel write entries in the *local* routing table, which the node then tried to delete as strays; - **the route server never routed its pool's own block.** An address on a point-to-point interface does not route the rest of its prefix, so a route server holding `10.66.0.1/16` answered "network is unreachable" for every node in the pool. That is merged 4b code doing exactly what it was written to do. The route server's command transport is now injectable so the harness can run its real commands in a namespace rather than over SSH; the orchestration on the node sits behind a `NetOps` trait so what the node *decides* is testable without root, with the kernel implementation proved by the harness. --- API_CHANGELOG.md | 4 +- Cargo.lock | 333 +++++- docs/agents/e2e-tests.md | 48 + lnvps_api/src/api/marketplace.rs | 4 - lnvps_api/src/provisioner/tunnel.rs | 57 +- lnvps_api/src/router/linux_ssh.rs | 26 +- lnvps_api/src/worker.rs | 8 +- lnvps_e2e/Cargo.toml | 7 + lnvps_e2e/tests/tunnel_netns.rs | 424 ++++++++ lnvps_node/Cargo.toml | 7 + lnvps_node/src/api.rs | 2 - lnvps_node/src/control.rs | 82 +- lnvps_node/src/lib.rs | 5 +- lnvps_node/src/main.rs | 68 +- lnvps_node/src/net.rs | 1475 +++++++++++++++------------ lnvps_node/src/net/tests.rs | 509 +++++++++ lnvps_node/src/netns.rs | 217 ++++ lnvps_node/tests/control_https.rs | 8 +- scripts/tunnel-e2e.sh | 43 + work/marketplace.md | 35 +- 20 files changed, 2578 insertions(+), 784 deletions(-) create mode 100644 lnvps_e2e/tests/tunnel_netns.rs create mode 100644 lnvps_node/src/net/tests.rs create mode 100644 lnvps_node/src/netns.rs create mode 100755 scripts/tunnel-e2e.sh diff --git a/API_CHANGELOG.md b/API_CHANGELOG.md index 79be4603..6bae5292 100644 --- a/API_CHANGELOG.md +++ b/API_CHANGELOG.md @@ -56,7 +56,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added -- **A node can fetch its whole data plane in one document** — `GET /api/v1/node/dataplane` (node token) returns the tunnel it already gets from `/node/tunnel`, plus the bridge guests are placed on, the gateway addresses the node must answer for, and the guests assigned to it (address, gateway, MAC). One call rather than three because the node applies these together or not at all: a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list — an address not in it is not that node's to send from. The gateway is the one the guest was actually configured with, taken from its IP range: it belongs to the range rather than to the node, and the guest believes it is on-link, so the node has to answer for it rather than invent one. +- **The route server now routes its tunnel pool's own blocks.** An address on a point-to-point interface does not route the rest of its prefix, so a route server holding `10.66.0.1/16` answered "network is unreachable" for every node in that pool — the peers were configured correctly and unreachable. Found by an end-to-end harness that builds both ends of a tunnel and sends real packets, not by review. + +- **A node can fetch its whole data plane in one document** — `GET /api/v1/node/dataplane` (node token) returns the tunnel it already gets from `/node/tunnel`, plus the gateway addresses the node must answer for, and the guests assigned to it (address, gateway, MAC). One call rather than three because the node applies these together or not at all: a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list — an address not in it is not that node's to send from. The gateway is the one the guest was actually configured with, taken from its IP range: it belongs to the range rather than to the node, and the guest believes it is on-link, so the node has to answer for it rather than invent one. - **A node takes one address, not a point-to-point link** — `address4`/`address6` in the tunnel response are now a `/32` and a `/128`, and `gateway4`/`gateway6` are one address shared by every node on the pool rather than a per-node link address. WireGuard is layer 3 and point-to-point: the node needs no gateway on its own side (`ip route add default dev wg0` suffices), so a `/31` spent two addresses describing something that needs one — and forced the route server to carry one address per node on a single interface, thousands of them on a /16 pool. Pool capacity is reported accordingly: a /24 places 253 nodes (256 less the block's network address, the route server's address after it, and the broadcast address) where it previously reported 128 links. diff --git a/Cargo.lock b/Cargo.lock index 244d3a0f..5efc91dd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -195,7 +195,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7af50940b73bf4e16c15c448a2b121c63f2d68e3e54b6a8731673cb4aa0cdff5" dependencies = [ "base64ct", - "blake2", + "blake2 0.11.0-rc.6", "cpufeatures 0.3.0", "password-hash 0.6.1", ] @@ -737,6 +737,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest 0.10.7", +] + [[package]] name = "blake2" version = "0.11.0-rc.6" @@ -1308,6 +1317,20 @@ dependencies = [ "subtle", ] +[[package]] +name = "curve25519-dalek" +version = "4.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "curve25519-dalek-derive", + "fiat-crypto 0.2.9", + "rustc_version", + "subtle", +] + [[package]] name = "curve25519-dalek" version = "5.0.0" @@ -1318,7 +1341,7 @@ dependencies = [ "cpufeatures 0.3.0", "curve25519-dalek-derive", "digest 0.11.3", - "fiat-crypto", + "fiat-crypto 0.3.0", "rand_core 0.10.1", "rustc_version", "subtle", @@ -1409,6 +1432,55 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b" +[[package]] +name = "defguard_boringtun" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d920cd0791b2199308a3c8cc0d41b88d96e4c07d70e98b658a4ee72ad32ca7e" +dependencies = [ + "aead 0.5.2", + "base64 0.22.1", + "blake2 0.10.6", + "chacha20poly1305", + "hex", + "hmac 0.12.1", + "ip_network", + "ip_network_table", + "libc", + "nix 0.31.2", + "parking_lot", + "ring", + "socket2 0.6.4", + "thiserror 2.0.18", + "tracing", + "x25519-dalek 2.0.1", +] + +[[package]] +name = "defguard_wireguard_rs" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a09896853b7e5f2302c3e6c6786b3dc0433bc092cc9e3e647812d617d0c2a0ce" +dependencies = [ + "base64 0.22.1", + "defguard_boringtun", + "ipnet", + "libc", + "log 0.4.32", + "netlink-packet-core", + "netlink-packet-generic", + "netlink-packet-route 0.31.0", + "netlink-packet-utils", + "netlink-packet-wireguard", + "netlink-sys", + "regex", + "serde", + "thiserror 2.0.18", + "windows", + "wireguard-nt", + "x25519-dalek 3.0.0", +] + [[package]] name = "delegate" version = "0.13.5" @@ -1630,7 +1702,7 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ebaa1a2bf1290ab3bfe5a7b771d050ebffab2711c19a81691c683a5144a25de" dependencies = [ - "curve25519-dalek", + "curve25519-dalek 5.0.0", "ed25519", "rand_core 0.10.1", "serde", @@ -1907,6 +1979,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "fiat-crypto" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" + [[package]] name = "fiat-crypto" version = "0.3.0" @@ -2021,9 +2099,9 @@ dependencies = [ [[package]] name = "futures-channel" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" dependencies = [ "futures-core", "futures-sink", @@ -2031,9 +2109,9 @@ dependencies = [ [[package]] name = "futures-core" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" [[package]] name = "futures-executor" @@ -2059,15 +2137,15 @@ dependencies = [ [[package]] name = "futures-io" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" [[package]] name = "futures-macro" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" dependencies = [ "proc-macro2", "quote", @@ -2076,15 +2154,15 @@ dependencies = [ [[package]] name = "futures-sink" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" [[package]] name = "futures-task" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" [[package]] name = "futures-timer" @@ -2094,9 +2172,9 @@ checksum = "af43fadb8a98512d547e37b4e92e0ced13e205c061b87b4623eff01d918d6968" [[package]] name = "futures-util" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" dependencies = [ "futures-channel", "futures-core", @@ -2889,6 +2967,28 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "ip_network" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aa2f047c0a98b2f299aa5d6d7088443570faae494e9ae1305e48be000c9e0eb1" + +[[package]] +name = "ip_network_table" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4099b7cfc5c5e2fe8c5edf3f6f7adf7a714c9cc697534f63a5a5da30397cb2c0" +dependencies = [ + "ip_network", + "ip_network_table-deps-treebitmap", +] + +[[package]] +name = "ip_network_table-deps-treebitmap" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e537132deb99c0eb4b752f0346b6a836200eaaa3516dd7e5514b63930a09e5d" + [[package]] name = "ipconfig" version = "0.3.4" @@ -2896,7 +2996,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d40460c0ce33d6ce4b0630ad68ff63d6661961c48b6dba35e5a4d81cfb48222" dependencies = [ "socket2 0.6.4", - "widestring", + "widestring 1.2.1", "windows-registry", "windows-result", "windows-sys 0.61.2", @@ -3275,6 +3375,16 @@ version = "0.2.183" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + [[package]] name = "libm" version = "0.2.16" @@ -3507,7 +3617,7 @@ dependencies = [ "uuid", "virt", "wiremock", - "x25519-dalek", + "x25519-dalek 3.0.0", ] [[package]] @@ -3555,6 +3665,10 @@ dependencies = [ "chrono", "futures-util", "hex", + "lnvps_api", + "lnvps_api_common", + "lnvps_node", + "nix 0.30.1", "nostr 0.44.3", "p256 0.13.2", "rand_core 0.6.4", @@ -3564,6 +3678,7 @@ dependencies = [ "serde_json", "sha2 0.10.9", "sqlx", + "tempfile", "tokio", "tokio-tungstenite 0.28.0", "webauthn-rs", @@ -3605,21 +3720,28 @@ name = "lnvps_node" version = "0.4.7" dependencies = [ "anyhow", + "async-trait", "axum", "axum-server", "base64 0.22.1", "clap", "config", + "defguard_wireguard_rs", "env_logger", + "futures-util", "hex", "http", "if-addrs", + "ipnetwork", "lnvps_host_util", "log 0.4.32", + "netlink-packet-route 0.30.0", + "nix 0.30.1", "nostr 0.44.3", "rand 0.9.4", "rcgen", "reqwest 0.12.28", + "rtnetlink", "rustls", "serde", "serde_json", @@ -3627,7 +3749,7 @@ dependencies = [ "tokio", "tokio-rustls", "tower", - "x25519-dalek", + "x25519-dalek 3.0.0", ] [[package]] @@ -3879,6 +4001,112 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0efe882e02d206d8d279c20eb40e03baf7cb5136a1476dc084a324fbc3ec42d" +[[package]] +name = "netlink-packet-core" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b897d7bd4f0af82e68d40d0344cf37e97f9c97ddf74a098de3e4da05e96ca395" +dependencies = [ + "paste", +] + +[[package]] +name = "netlink-packet-generic" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f891b2e0054cac5a684a06628f59568f841c93da4e551239da6e518f539e775" +dependencies = [ + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-route" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be8919612f6028ab4eacbbfe1234a9a43e3722c6e0915e7ff519066991905092" +dependencies = [ + "bitflags", + "libc", + "log 0.4.32", + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-route" +version = "0.31.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2288fcb784eb3defd5fb16f4c4160d5f477de192eac730f43e1d11c24d9a007" +dependencies = [ + "bitflags", + "libc", + "log 0.4.32", + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-utils" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3176f18d11a1ae46053e59ec89d46ba318ae1343615bd3f8c908bfc84edae35c" +dependencies = [ + "byteorder", + "pastey", + "thiserror 2.0.18", +] + +[[package]] +name = "netlink-packet-wireguard" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8901a50aa367f28d70e8c2a6b379f51efef1e5acebbe05f940ae00363d7d83fd" +dependencies = [ + "bitflags", + "libc", + "log 0.4.32", + "netlink-packet-core", + "netlink-packet-generic", +] + +[[package]] +name = "netlink-proto" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6f7398dddf5f152d2a91a2921a134c6097056e292c0d4b9906007855e7cece6" +dependencies = [ + "bytes", + "futures-channel", + "futures-util", + "log 0.4.32", + "netlink-packet-core", + "netlink-sys", + "thiserror 2.0.18", +] + +[[package]] +name = "netlink-sys" +version = "0.8.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd6c30ed10fa69cc491d491b85cc971f6bdeb8e7367b7cde2ee6cc878d583fae" +dependencies = [ + "bytes", + "futures-util", + "libc", + "log 0.4.32", + "tokio", +] + +[[package]] +name = "nix" +version = "0.30.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] + [[package]] name = "nix" version = "0.31.2" @@ -4262,7 +4490,7 @@ checksum = "d17b851e6b3e378ab4ecb07fa2ed23f4d15f075735f8fec9fa1e7bdce5f8301f" dependencies = [ "ecdsa 0.17.0", "elliptic-curve 0.14.1", - "fiat-crypto", + "fiat-crypto 0.3.0", "primefield", "primeorder 0.14.0", "sha2 0.11.0", @@ -4365,6 +4593,12 @@ dependencies = [ "phc", ] +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + [[package]] name = "pastey" version = "0.1.1" @@ -5335,6 +5569,24 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rtnetlink" +version = "0.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc19f84f710fa2f337617f9bc0400260a94224bde7bae28fd8879f3771ca5784" +dependencies = [ + "futures-channel", + "futures-util", + "log 0.4.32", + "netlink-packet-core", + "netlink-packet-route 0.30.0", + "netlink-proto", + "netlink-sys", + "nix 0.30.1", + "thiserror 1.0.69", + "tokio", +] + [[package]] name = "russh" version = "0.62.4" @@ -5351,7 +5603,7 @@ dependencies = [ "cipher 0.5.2", "crypto-bigint 0.7.5", "ctr 0.10.1", - "curve25519-dalek", + "curve25519-dalek 5.0.0", "data-encoding", "delegate", "der 0.8.1", @@ -5414,7 +5666,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3aec6cb630dbe85d72ffd7bcd95f07e1bd69f9f270ee8adfa1afe443a6331438" dependencies = [ "log 0.4.32", - "nix", + "nix 0.31.2", "ssh-encoding 0.3.0", "windows-sys 0.61.2", ] @@ -7478,6 +7730,12 @@ dependencies = [ "wasite", ] +[[package]] +name = "widestring" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c168940144dd21fd8046987c16a46a33d5fc84eec29ef9dcddc2ac9e31526b7c" + [[package]] name = "widestring" version = "1.2.1" @@ -7854,6 +8112,22 @@ dependencies = [ "memchr", ] +[[package]] +name = "wireguard-nt" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22b4dbcc6c93786cf22e420ef96e8976bfb92a455070282302b74de5848191f4" +dependencies = [ + "bitflags", + "getrandom 0.2.17", + "ipnet", + "libloading", + "log 0.4.32", + "thiserror 1.0.69", + "widestring 0.4.3", + "windows-sys 0.59.0", +] + [[package]] name = "wiremock" version = "0.6.5" @@ -7900,13 +8174,24 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" +[[package]] +name = "x25519-dalek" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7e468321c81fb07fa7f4c636c3972b9100f0346e5b6a9f2bd0603a52f7ed277" +dependencies = [ + "curve25519-dalek 4.1.3", + "rand_core 0.6.4", +] + [[package]] name = "x25519-dalek" version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e7e8131a03190127fb2263afc72b322ecadae46b6ff8c6f399ff5d02f5559af6" dependencies = [ - "curve25519-dalek", + "curve25519-dalek 5.0.0", + "getrandom 0.4.3", "rand_core 0.10.1", "zeroize", ] diff --git a/docs/agents/e2e-tests.md b/docs/agents/e2e-tests.md index de9d56bd..7b901314 100644 --- a/docs/agents/e2e-tests.md +++ b/docs/agents/e2e-tests.md @@ -315,3 +315,51 @@ The `.github/workflows/e2e.yml` workflow runs E2E tests on every pull request. I | `.github/e2e/api-config.yaml` | User API config template (DB URL replaced at runtime) | | `.github/e2e/admin-config.yaml` | Admin API config template (DB URL replaced at runtime) | | `.github/e2e/wait-for-lnd.sh` | Script to wait for LND readiness and mine initial blocks | + +## Marketplace tunnel harness (`tests/tunnel_netns.rs`) + +A second kind of end-to-end test lives in the same crate and shares none of the +above infrastructure: no API server, no database, no docker. It builds **both +ends of a marketplace tunnel** out of Linux network namespaces and sends real +packets across it. + +```text + [rs netns] [machine netns] [lnvps netns] [guest netns] + wgln <══ WireGuard ══> wg0 created here, then ══> wg0 veth + 10.66.0.1/24 its UDP socket stays here 10.66.0.2/32 br-lnvps ── 203.0.113.5/24 +``` + +Both ends run production code: the route server is configured through +`LinuxSshRouter` with its command transport pointed at `ip netns exec` instead +of SSH, and the node end is `lnvps_node::net::apply` — the same netlink calls a +real node makes. + +```sh +sudo ./scripts/tunnel-e2e.sh # both scenarios +./scripts/tunnel-e2e.sh --filter a_guest_behind # one +``` + +Requires **root** (namespaces, veth, WireGuard) and `wireguard-tools` for the +route-server end, so the tests are `#[ignore]`d and only run from that script. + +**Why it exists.** Unit tests assert what the code *decides* — which commands +the route server issues, which netlink operations the node performs. None of +that proves a packet moves. This harness caught four things nothing else did: + +- the node's namespace was pinned from `/proc/self/ns/net`, which in a + multi-threaded process is the *process's* namespace, so every "isolated" + interface was silently landing in the operator's own network; +- WireGuard's netlink calls ran outside the namespace the interface had been + moved into, reporting "no such device" about an interface that plainly existed; +- addresses on an interface produce routes in the kernel's *local* table, which + the node then tried to delete as strays; +- **the route server never routed the pool's own block.** An address on a + point-to-point interface does not route the rest of its prefix, so a route + server holding `10.66.0.1/16` answered "network is unreachable" for every node + in the pool. Not visible in any unit test, because the code did exactly what it + was written to do. + +Coverage note: the netlink implementation (`lnvps_node::net::kernel`) and +`lnvps_node::netns` are exercised here rather than by the normal test run, the +same way `lnvps_fw`'s datapath is covered by its netns harness. Measure them +with `sudo -E cargo llvm-cov -p lnvps_node -- --include-ignored`. diff --git a/lnvps_api/src/api/marketplace.rs b/lnvps_api/src/api/marketplace.rs index 171126b3..75387689 100644 --- a/lnvps_api/src/api/marketplace.rs +++ b/lnvps_api/src/api/marketplace.rs @@ -614,9 +614,6 @@ async fn v1_node_get_tunnel( #[derive(Serialize, Debug)] pub struct ApiNodeDataPlane { pub tunnel: ApiNodeTunnel, - /// The bridge guests are placed on. LNVPS decides the name so every node is - /// the same shape. - pub bridge: String, /// Gateway addresses this node must answer for on the bridge. They belong /// to the ranges the guests were addressed from, and the guests believe /// they are on-link. @@ -640,7 +637,6 @@ impl From for ApiNodeDataPlane { fn from(d: crate::provisioner::NodeDataPlane) -> Self { Self { gateways: d.gateways(), - bridge: d.bridge.clone(), guests: d .guests .iter() diff --git a/lnvps_api/src/provisioner/tunnel.rs b/lnvps_api/src/provisioner/tunnel.rs index c39d1d4d..0d37501d 100644 --- a/lnvps_api/src/provisioner/tunnel.rs +++ b/lnvps_api/src/provisioner/tunnel.rs @@ -371,6 +371,18 @@ pub async fn plan_pool(db: &Arc, pool: &TunnelPool) -> Result, pool: &TunnelPool) -> Result, } @@ -466,7 +480,6 @@ pub async fn node_dataplane( Ok(Some(NodeDataPlane { guests: node_guests(db, node).await?, tunnel, - bridge: NODE_BRIDGE.to_string(), })) } @@ -1062,12 +1075,27 @@ mod tests { plan.addresses, vec!["10.66.0.1/24".to_string(), "fd00:66::1/64".to_string()] ); + // ...the pool's own blocks, because an address on a point-to-point + // interface does not route the rest of its prefix, and every node in + // the pool lives in it... + assert!( + plan.routes.contains(&"10.66.0.0/24".to_string()), + "{plan:?}" + ); + assert!( + plan.routes.contains(&"fd00:66::/64".to_string()), + "{plan:?}" + ); // ...and a route for each guest address, because AllowedIPs picks the // peer for a packet already headed down the tunnel, it does not put it // there. - assert_eq!( - plan.routes, - vec!["2001:db8::5/128".to_string(), "203.0.113.5/32".to_string()] + assert!( + plan.routes.contains(&"2001:db8::5/128".to_string()), + "{plan:?}" + ); + assert!( + plan.routes.contains(&"203.0.113.5/32".to_string()), + "{plan:?}" ); assert_eq!(allocation.tunnel.pool_id, Some(pool_id)); } @@ -1094,7 +1122,7 @@ mod tests { } let pool = db.get_tunnel_pool(pool_id).await.unwrap(); let plan = plan_pool(&db, &pool).await.unwrap(); - assert_eq!(plan.routes, Vec::::new()); + assert!(!plan.routes.iter().any(|r| r.starts_with("203.0.113"))); assert_eq!(plan.peers[0].allowed_ips.len(), 2, "only the node's own"); // A deleted VM takes its addressing with it for the same reason. @@ -1109,7 +1137,7 @@ mod tests { } } let plan = plan_pool(&db, &pool).await.unwrap(); - assert_eq!(plan.routes, Vec::::new()); + assert!(!plan.routes.iter().any(|r| r.starts_with("203.0.113"))); } /// A tunnel that cannot be realised contributes nothing at all, rather than @@ -1134,7 +1162,9 @@ mod tests { db.update_tunnel(&broken).await.unwrap(); let plan = plan_pool(&db, &pool).await.unwrap(); assert!(plan.peers.is_empty()); - assert!(plan.routes.is_empty()); + // The pool's blocks are routed whether or not anything is placed + // in it; the interface exists either way. + assert_eq!(plan.routes.len(), 2); // The interface still holds the pool's address: it exists whether // or not anything has been placed in it yet. assert_eq!(plan.addresses.len(), 2); @@ -1165,7 +1195,8 @@ mod tests { let plan = plan_pool(&db, &pool).await.unwrap(); assert_eq!(plan.peers.len(), 1); assert_eq!(plan.peers[0].allowed_ips, vec!["10.66.9.1/32".to_string()]); - assert!(plan.routes.is_empty()); + // The pool's own blocks, and nothing a guest brought. + assert_eq!(plan.routes.len(), 2); let _ = mock; } @@ -1186,7 +1217,6 @@ mod tests { add_guest(&db, &mock, host.id, &["203.0.113.5", "2001:db8::5"]).await; let plane = node_dataplane(&db, &node).await.unwrap().unwrap(); - assert_eq!(plane.bridge, NODE_BRIDGE); assert_eq!( plane.tunnel.tunnel.address4.as_deref(), Some("10.66.0.2/32") @@ -1227,7 +1257,6 @@ mod tests { let plane = node_dataplane(&db, &node).await.unwrap().unwrap(); assert!(plane.guests.is_empty()); assert!(plane.gateways().is_empty()); - assert_eq!(plane.bridge, NODE_BRIDGE); } /// Give `host` a VM holding `ips`. Written through the mock's maps because diff --git a/lnvps_api/src/router/linux_ssh.rs b/lnvps_api/src/router/linux_ssh.rs index 00c8ac91..c5b0e119 100644 --- a/lnvps_api/src/router/linux_ssh.rs +++ b/lnvps_api/src/router/linux_ssh.rs @@ -28,18 +28,18 @@ pub struct LinuxSshRouter { interface: String, /// SSH private key (PEM) key: String, - /// Stands in for the SSH connection under test. + /// Where commands are run, when that is not an SSH session. /// - /// These methods run commands as root on somebody else's route server, so - /// what is worth asserting is the exact command issued — which needs the - /// transport replaced, not mocked around. - #[cfg(test)] + /// These methods run commands as root on a route server, so what is worth + /// asserting is the exact command issued — which needs the transport + /// replaced rather than mocked around. Tests record the commands; the + /// end-to-end harness runs them in a network namespace against a real + /// kernel. exec: Option, } /// A stand-in for running a command over SSH. -#[cfg(test)] -type ExecFn = std::sync::Arc OpResult + Send + Sync>; +pub type ExecFn = std::sync::Arc OpResult + Send + Sync>; impl LinuxSshRouter { /// Build a router from the stored config `url` and `token`. @@ -68,16 +68,17 @@ impl LinuxSshRouter { username, interface, key: key.to_string(), - #[cfg(test)] exec: None, }) } - /// A router whose commands are handled by `exec` instead of an SSH session. - #[cfg(test)] - fn with_exec(exec: ExecFn) -> Self { + /// A router whose commands are run by `exec` rather than over SSH. + /// + /// The transport is the only thing that changes: the commands, and the + /// decisions behind them, are the ones a real route server gets. + pub fn with_exec(exec: ExecFn) -> Self { Self { - host: "test:22".to_string(), + host: "local".to_string(), username: "root".to_string(), interface: "eth0".to_string(), key: String::new(), @@ -101,7 +102,6 @@ impl LinuxSshRouter { /// Run a command, mapping connection failures to transient errors and /// non-zero exits to fatal errors. Returns stdout on success. async fn exec_checked(&self, cmd: &str) -> OpResult { - #[cfg(test)] if let Some(exec) = &self.exec { return exec(cmd); } diff --git a/lnvps_api/src/worker.rs b/lnvps_api/src/worker.rs index 669728ac..917a7b1b 100644 --- a/lnvps_api/src/worker.rs +++ b/lnvps_api/src/worker.rs @@ -5295,11 +5295,13 @@ mod tests { vec!["10.66.0.1/24".to_string()] ); // AllowedIPs picks which peer a packet belongs to; it does not put the - // packet on the tunnel. Without this route the guest's return traffic - // is dropped as unroutable. + // packet on the tunnel. Without these routes the guest's return traffic + // is dropped as unroutable — and without the pool's own block, so is + // everything addressed to the nodes themselves, because an address on a + // point-to-point interface does not route the rest of its prefix. assert_eq!( mr.interface_routes(&interface).await, - vec!["203.0.113.5/32".to_string()] + vec!["10.66.0.0/24".to_string(), "203.0.113.5/32".to_string()] ); assert_eq!(tunnel.pool_id, Some(pool_id)); diff --git a/lnvps_e2e/Cargo.toml b/lnvps_e2e/Cargo.toml index 34d3f17f..f1d4e72b 100644 --- a/lnvps_e2e/Cargo.toml +++ b/lnvps_e2e/Cargo.toml @@ -32,6 +32,13 @@ tokio-tungstenite = "0.28" futures-util = "0.3" [dev-dependencies] +# Both ends of a marketplace tunnel, driven by the real code, in network +# namespaces: the route server's configuration path and the node daemon's. +lnvps_api = { path = "../lnvps_api" } +lnvps_api_common = { path = "../lnvps_api_common" } +lnvps_node = { path = "../lnvps_node" } +tempfile = "3" +nix = { version = "0.30", features = ["user"] } # Used only to validate the software authenticator offline: register via # webauthn-rs-core (as the server does) and verify a discoverable assertion via # the high-level webauthn-rs discoverable flow. diff --git a/lnvps_e2e/tests/tunnel_netns.rs b/lnvps_e2e/tests/tunnel_netns.rs new file mode 100644 index 00000000..bff630b7 --- /dev/null +++ b/lnvps_e2e/tests/tunnel_netns.rs @@ -0,0 +1,424 @@ +//! Both ends of a marketplace tunnel, on a real kernel, carrying real packets. +//! +//! Everything below this line has been proved by unit tests that assert what +//! the code *decides*: which commands the route server issues, which netlink +//! operations the node performs. None of that proves a packet moves. This +//! harness builds the two ends out of network namespaces and pings across the +//! tunnel — first the node itself, then a guest sitting behind it, which is the +//! path a customer's traffic actually takes. +//! +//! ```text +//! [rs netns] [test machine's netns] [lnvps netns] [guest netns] +//! wgln <══ WireGuard ══> wg0 created here, then ══> wg0 veth +//! 10.66.0.1/24 its UDP socket stays here 10.66.0.2/32 br-lnvps ── 203.0.113.5/24 +//! │ 203.0.113.1/32 +//! rs_up veth ────────────────── node_up veth +//! 198.51.100.1/24 198.51.100.2/24 +//! ``` +//! +//! The shape is the production one, including the part that is easy to get +//! wrong: `wg0` is created in the machine's own namespace so its UDP socket can +//! reach the route server through the operator's uplink, and is then moved into +//! the LNVPS namespace so that everything carried *over* the tunnel is isolated +//! from the operator's network. +//! +//! Requires root (namespaces, veths, WireGuard) so it is `#[ignore]`d; run it +//! with `scripts/tunnel-e2e.sh`. + +use std::process::Command; +use std::sync::Arc; + +use anyhow::{Context, Result, bail}; +use lnvps_api::router::{Tunnel, TunnelConfig, TunnelRouter, WireguardConfig, WireguardPeer}; +use lnvps_node::net::{DesiredDataPlane, DesiredGuest, DesiredTunnel}; + +/// Underlay: the "internet" between the two machines. +const RS_UNDERLAY: &str = "198.51.100.1/24"; +const NODE_UNDERLAY: &str = "198.51.100.2/24"; +/// Inner tunnel addresses, as the allocator hands them out. +const RS_INNER: &str = "10.66.0.1/24"; +const NODE_INNER: &str = "10.66.0.2/32"; +/// A customer address, as LNVPS assigns it to a guest on this node. +const GUEST_ADDRESS: &str = "203.0.113.5"; +const GUEST_GATEWAY: &str = "203.0.113.1"; + +/// Namespaces, torn down on drop even when a test panics. +struct Topology { + rs: String, + guest: String, + /// The node's data plane namespace, pinned where iproute2 looks so an + /// operator — and this harness — can reach it with `ip netns exec`. + dataplane: String, +} + +impl Topology { + fn new(tag: &str) -> Result { + let topology = Self { + rs: format!("lnvps-e2e-rs-{tag}"), + guest: format!("lnvps-e2e-guest-{tag}"), + dataplane: format!("lnvps-e2e-dp-{tag}"), + }; + topology.teardown(); + + run("ip", &["netns", "add", &topology.rs])?; + run("ip", &["netns", "add", &topology.guest])?; + + // The underlay: the route server and the node's machine, reachable to + // each other and to nothing else. + run( + "ip", + &[ + "link", "add", "e2e-rs", "type", "veth", "peer", "name", "e2e-node", + ], + )?; + run("ip", &["link", "set", "e2e-rs", "netns", &topology.rs])?; + topology.in_rs(&["ip", "addr", "add", RS_UNDERLAY, "dev", "e2e-rs"])?; + topology.in_rs(&["ip", "link", "set", "e2e-rs", "up"])?; + topology.in_rs(&["ip", "link", "set", "lo", "up"])?; + run("ip", &["addr", "add", NODE_UNDERLAY, "dev", "e2e-node"])?; + run("ip", &["link", "set", "e2e-node", "up"])?; + + Ok(topology) + } + + fn in_rs(&self, argv: &[&str]) -> Result { + let mut full = vec!["netns", "exec", &self.rs]; + full.extend_from_slice(argv); + run("ip", &full) + } + + fn in_guest(&self, argv: &[&str]) -> Result { + let mut full = vec!["netns", "exec", &self.guest]; + full.extend_from_slice(argv); + run("ip", &full) + } + + /// Run commands in the node's *data plane* namespace, which the production + /// code created and pinned. + fn in_dataplane(&self, argv: &[&str]) -> Result { + let mut full = vec!["netns", "exec", &self.dataplane]; + full.extend_from_slice(argv); + run("ip", &full) + } + + /// The namespace the node's code builds, as production code would. + fn open_dataplane(&self) -> Result { + lnvps_node::netns::ensure(std::path::Path::new("/run/netns"), &self.dataplane) + } + + fn teardown(&self) { + let _ = Command::new("ip") + .args(["netns", "delete", &self.rs]) + .output(); + let _ = Command::new("ip") + .args(["netns", "delete", &self.guest]) + .output(); + let _ = Command::new("ip") + .args(["netns", "delete", &self.dataplane]) + .output(); + let _ = Command::new("ip") + .args(["link", "del", "e2e-node"]) + .output(); + let _ = Command::new("ip").args(["link", "del", "wg0"]).output(); + } +} + +impl Drop for Topology { + fn drop(&mut self) { + self.teardown(); + } +} + +fn run(program: &str, args: &[&str]) -> Result { + let out = Command::new(program) + .args(args) + .output() + .with_context(|| format!("cannot run {program}"))?; + if !out.status.success() { + bail!( + "`{program} {}` failed: {}", + args.join(" "), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(String::from_utf8_lossy(&out.stdout).to_string()) +} + +/// A route server whose commands run inside a namespace instead of over SSH. +/// +/// The commands, and every decision behind them, are the ones a real route +/// server is given; only the transport changes. +fn route_server(namespace: &str) -> lnvps_api::router::LinuxSshRouter { + let namespace = namespace.to_string(); + lnvps_api::router::LinuxSshRouter::with_exec(Arc::new(move |cmd: &str| { + let out = Command::new("ip") + .args(["netns", "exec", &namespace, "sh", "-c", cmd]) + .output() + .map_err(|e| lnvps_api_common::retry::OpError::Fatal(e.into()))?; + if !out.status.success() { + return Err(lnvps_api_common::retry::OpError::Fatal(anyhow::anyhow!( + "`{cmd}` failed: {}", + String::from_utf8_lossy(&out.stderr).trim() + ))); + } + Ok(String::from_utf8_lossy(&out.stdout).to_string()) + })) +} + +/// Skip rather than fail when the machine cannot run this at all: a developer's +/// laptop without root should not report a red test it was never able to run. +fn requirements_met() -> bool { + if !nix::unistd::Uid::effective().is_root() { + eprintln!("skipping: needs root for network namespaces and WireGuard"); + return false; + } + if run("modprobe", &["wireguard"]).is_err() + && !std::path::Path::new("/sys/module/wireguard").exists() + { + eprintln!("skipping: no WireGuard support in this kernel"); + return false; + } + true +} + +/// The whole path: LNVPS's route server, the node's daemon code, and a guest +/// behind it — with packets crossing all of it. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires root and network namespaces; run with scripts/tunnel-e2e.sh"] +async fn a_guest_behind_a_node_is_reachable_from_the_route_server() -> Result<()> { + if !requirements_met() { + return Ok(()); + } + let topology = Topology::new("guest")?; + + // ---- the node generates its own key; LNVPS never sees the private half + let state_dir = tempfile::tempdir()?; + let node_key = lnvps_node::wgkey::load_or_generate(state_dir.path())?; + + // ---- the route server's own interface, configured by production code + let server_key = lnvps_api_common::generate_wireguard_keypair()?; + let rs = route_server(&topology.rs); + let interface = "wgln1"; + rs.add_tunnel(&Tunnel { + id: None, + name: interface.to_string(), + local_addr: None, + remote_addr: None, + enabled: true, + config: TunnelConfig::Wireguard(WireguardConfig { + listen_port: Some(51820), + private_key: Some(server_key.private_key.clone()), + public_key: Some(lnvps_api_common::wireguard_key_to_base64( + &server_key.public_key, + )), + peers: vec![], + }), + }) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + + // The pool's address, and the peer as the reconciler builds it: the node's + // own address plus exactly the guest addresses LNVPS assigned to it, which + // is the anti-spoof boundary. + rs.sync_tunnel_addresses(interface, &[RS_INNER.to_string()]) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + rs.set_tunnel_peer( + interface, + &WireguardPeer { + public_key: node_key.public_base64(), + endpoint: None, + allowed_ips: vec![NODE_INNER.to_string(), format!("{GUEST_ADDRESS}/32")], + persistent_keepalive: None, + }, + ) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + // The pool's block *and* the guest address: an address on a + // point-to-point interface does not route the rest of its prefix, so + // without the block the route server cannot reach any node in the pool. + rs.sync_tunnel_routes( + interface, + &["10.66.0.0/24".to_string(), format!("{GUEST_ADDRESS}/32")], + ) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + + // ---- the node applies the document LNVPS would have sent it + let kernel = lnvps_node::net::Kernel::in_namespace(topology.open_dataplane()?)?; + let desired = DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some(NODE_INNER.to_string()), + address6: None, + gateway4: Some("10.66.0.1".to_string()), + gateway6: None, + server_public_key: hex::encode(&server_key.public_key), + endpoint: "198.51.100.1:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + gateways: vec![GUEST_GATEWAY.to_string()], + guests: vec![DesiredGuest { + address: format!("{GUEST_ADDRESS}/32"), + gateway: GUEST_GATEWAY.to_string(), + mac: None, + }], + }; + lnvps_node::net::apply(&kernel, &desired, &node_key).await?; + + // ---- a guest on the node's bridge, addressed as a customer's VM is + run( + "ip", + &[ + "link", + "add", + "e2e-guest", + "type", + "veth", + "peer", + "name", + "e2e-tap", + ], + )?; + run( + "ip", + &["link", "set", "e2e-tap", "netns", &topology.dataplane], + )?; + topology.in_dataplane(&["ip", "link", "set", "e2e-tap", "master", "br-lnvps"])?; + topology.in_dataplane(&["ip", "link", "set", "e2e-tap", "up"])?; + run( + "ip", + &["link", "set", "e2e-guest", "netns", &topology.guest], + )?; + topology.in_guest(&[ + "ip", + "addr", + "add", + &format!("{GUEST_ADDRESS}/24"), + "dev", + "e2e-guest", + ])?; + topology.in_guest(&["ip", "link", "set", "e2e-guest", "up"])?; + // The guest is configured with its range's gateway and believes it is + // on-link — which is exactly why the node holds that address and answers + // for it. + topology.in_guest(&["ip", "route", "replace", "default", "via", GUEST_GATEWAY])?; + + // ---- the tunnel itself + let node_inner = NODE_INNER.split('/').next().unwrap(); + topology + .in_rs(&["ping", "-c", "3", "-W", "5", node_inner]) + .with_context(|| { + format!( + "the route server could not reach the node over the tunnel\n\ + route server:\n{}{}\n\ + node data plane:\n{}{}", + topology.in_rs(&["ip", "addr"]).unwrap_or_default(), + topology.in_rs(&["wg", "show"]).unwrap_or_default(), + topology.in_dataplane(&["ip", "addr"]).unwrap_or_default(), + topology.in_dataplane(&["wg", "show"]).unwrap_or_default(), + ) + })?; + + // ---- and the path a customer's traffic actually takes + topology + .in_rs(&["ping", "-c", "3", "-W", "5", GUEST_ADDRESS]) + .context("the route server could not reach a guest behind the node")?; + + // The node reports itself healthy only once that has happened: `wg0` comes + // up perfectly happily with a peer that never answers. + let state = lnvps_node::net::observe(&kernel).await?; + assert!(state.tunnel_up, "{state:?}"); + assert!(state.bridge_up, "{state:?}"); + assert!( + state.last_handshake_secs.is_some(), + "a tunnel that carried packets reported no handshake: {state:?}" + ); + assert!(state.healthy(), "{state:?}"); + Ok(()) +} + +/// The isolation the namespace exists for: the operator's own network is not +/// reachable from inside the data plane, and their default route is untouched. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires root and network namespaces; run with scripts/tunnel-e2e.sh"] +async fn the_operators_machine_keeps_its_own_network() -> Result<()> { + if !requirements_met() { + return Ok(()); + } + let topology = Topology::new("isolation")?; + let state_dir = tempfile::tempdir()?; + let node_key = lnvps_node::wgkey::load_or_generate(state_dir.path())?; + let server_key = lnvps_api_common::generate_wireguard_keypair()?; + + let default_before = run("ip", &["-j", "route", "show", "default"])?; + + let kernel = lnvps_node::net::Kernel::in_namespace(topology.open_dataplane()?)?; + lnvps_node::net::apply( + &kernel, + &DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some(NODE_INNER.to_string()), + address6: None, + gateway4: Some("10.66.0.1".to_string()), + gateway6: None, + server_public_key: hex::encode(&server_key.public_key), + endpoint: "198.51.100.1:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + gateways: vec![GUEST_GATEWAY.to_string()], + guests: vec![], + }, + &node_key, + ) + .await?; + + // The interfaces exist in the namespace and nowhere else. Asserted first, + // because if this is wrong every later assertion is wrong for the same + // reason and this is the one that says why. + assert!( + run("ip", &["link", "show", "wg0"]).is_err(), + "wg0 is still in the machine's namespace: {}", + run("ip", &["link", "show"]).unwrap_or_default() + ); + assert!( + topology + .in_dataplane(&["ip", "link", "show", "wg0"]) + .is_ok() + ); + assert!(run("ip", &["link", "show", "br-lnvps"]).is_err()); + + // The default route the node installed is the data plane's, not the + // machine's: taking an operator's default route would send their own + // traffic up a tunnel they do not own. + assert_eq!( + default_before, + run("ip", &["-j", "route", "show", "default"])?, + "the machine's default route changed" + ); + + // Forwarding is enabled in the namespace only. On a machine that is also + // the operator's workstation, turning it on globally is not ours to do. + let inside = topology.in_dataplane(&["cat", "/proc/sys/net/ipv4/ip_forward"]); + assert_eq!(inside.unwrap_or_default().trim(), "1"); + + assert!( + topology + .in_dataplane(&["ip", "link", "show", "e2e-node"]) + .is_err(), + "the operator's uplink is reachable from inside the data plane" + ); + Ok(()) +} + +/// LNVPS and the node hold the bridge name as a constant each. They are not +/// sent to each other precisely so they cannot disagree at runtime — which only +/// works if they agree at build time, and this is where both are in scope. +#[test] +fn both_ends_agree_on_the_bridge() { + assert_eq!( + lnvps_api::provisioner::NODE_BRIDGE, + lnvps_node::net::GUEST_BRIDGE + ); +} diff --git a/lnvps_node/Cargo.toml b/lnvps_node/Cargo.toml index 081d5e19..428d8c67 100644 --- a/lnvps_node/Cargo.toml +++ b/lnvps_node/Cargo.toml @@ -47,6 +47,13 @@ env_logger.workspace = true serde.workspace = true serde_json.workspace = true config.workspace = true +rtnetlink = "0.21" +defguard_wireguard_rs = "0.11" +futures-util = "0.3.33" +async-trait.workspace = true +ipnetwork.workspace = true +netlink-packet-route = "0.30" +nix = { version = "0.30", features = ["sched", "mount", "fs"] } [dev-dependencies] tempfile = "3" diff --git a/lnvps_node/src/api.rs b/lnvps_node/src/api.rs index 9fbc9cc2..819f4d1a 100644 --- a/lnvps_node/src/api.rs +++ b/lnvps_node/src/api.rs @@ -219,7 +219,6 @@ mod tests { "keepalive": 25, "mtu": 1420 }, - "bridge": "br-lnvps", "gateways": ["203.0.113.1"], "guests": [ {"address": "203.0.113.5/32", "gateway": "203.0.113.1", "mac": null} @@ -246,7 +245,6 @@ mod tests { calls[0] ); - assert_eq!(plane.bridge, "br-lnvps"); assert_eq!(plane.tunnel.mtu, 1420); assert_eq!(plane.guests.len(), 1); assert_eq!(plane.gateways, vec!["203.0.113.1".to_string()]); diff --git a/lnvps_node/src/control.rs b/lnvps_node/src/control.rs index 0fca4061..f80fa1a9 100644 --- a/lnvps_node/src/control.rs +++ b/lnvps_node/src/control.rs @@ -57,29 +57,31 @@ pub struct ControlState { /// is compared against this, so a request signed for one node cannot be /// replayed against another by setting `Host` to the first node's address. pub base_url: String, - /// The bridge guests are placed on, as LNVPS named it. Kept so the status - /// report observes the bridge this node was actually told to build rather - /// than one the daemon assumes. - pub bridge: String, + /// How the node reads its own network back. Held here so a test can supply + /// a machine that is not the one the tests run on. + pub net: Arc, } impl ControlState { /// Build state for a node serving on `addr`. - pub fn new(control_pubkey: PublicKey, addr: SocketAddr) -> Self { + pub fn new( + control_pubkey: PublicKey, + addr: SocketAddr, + net: Arc, + ) -> Self { Self { control_pubkey, replay: Mutex::new(ReplayGuard::new(REPLAY_WINDOW, REPLAY_CAPACITY)), base_url: format!("https://{addr}"), - bridge: crate::net::DEFAULT_BRIDGE.to_string(), + net, } } - /// Same, for a node whose data plane names a different bridge. - pub fn with_bridge(control_pubkey: PublicKey, addr: SocketAddr, bridge: &str) -> Self { - Self { - bridge: bridge.to_string(), - ..Self::new(control_pubkey, addr) - } + /// The data plane as this machine actually has it. + pub async fn observe(&self) -> crate::net::DataPlaneState { + crate::net::observe(self.net.as_ref()) + .await + .unwrap_or_default() } } @@ -110,6 +112,22 @@ pub fn router(state: Arc) -> Router { /// Serve the control API over HTTPS until the process exits. pub async fn serve(state: Arc, addr: SocketAddr, tls: NodeTls) -> Result<()> { + let listener = std::net::TcpListener::bind(addr) + .with_context(|| format!("Cannot bind the control API to {addr}"))?; + serve_on(state, listener, tls).await +} + +/// Serve on a socket somebody else opened. +/// +/// The tunnel address lives in the data plane's network namespace, so the +/// listening socket has to be created in there — a socket keeps the namespace +/// it was created in, which is what lets the rest of the daemon stay in the +/// machine's own namespace and go on reaching LNVPS. +pub async fn serve_on( + state: Arc, + listener: std::net::TcpListener, + tls: NodeTls, +) -> Result<()> { // rustls needs a process-wide crypto provider; installing twice is not an // error worth failing a startup over. let _ = rustls::crypto::ring::default_provider().install_default(); @@ -118,11 +136,17 @@ pub async fn serve(state: Arc, addr: SocketAddr, tls: NodeTls) -> .await .context("Control API TLS configuration is invalid")?; + let addr = listener + .local_addr() + .context("The control API socket has no address")?; log::info!( "Control API listening on https://{addr} (certificate fingerprint {})", tls.fingerprint ); - axum_server::bind_rustls(addr, cfg) + listener + .set_nonblocking(true) + .context("Cannot make the control API socket non-blocking")?; + axum_server::from_tcp_rustls(listener, cfg) .serve(router(state).into_make_service()) .await .context("Control API server stopped") @@ -202,8 +226,11 @@ async fn get_status(State(state): State>) -> Json // Observed on demand rather than cached: a cached answer is a report // that the tunnel was up once, which is exactly the thing the gate must // not accept. - dataplane: crate::net::observe(&crate::net::SystemCommands, &state.bridge) - .unwrap_or_default(), + // Observed on demand rather than cached: a cached answer is a report + // that the tunnel was up once, which is exactly what the health gate + // must not accept. A machine this cannot be read from reports nothing + // configured, which is a truthful answer and one the gate can act on. + dataplane: state.observe().await, }) } @@ -219,7 +246,11 @@ mod tests { const ADDR: &str = "10.66.0.7:8890"; fn state_for(keys: &Keys, addr: &str) -> Arc { - Arc::new(ControlState::new(keys.public_key(), addr.parse().unwrap())) + Arc::new(ControlState::new( + keys.public_key(), + addr.parse().unwrap(), + Arc::new(crate::net::tests::FakeKernel::default()), + )) } /// A NIP-98 header, with every field overridable so tests can tamper with @@ -267,22 +298,19 @@ mod tests { assert_eq!(code, StatusCode::OK); } - /// The status report observes the bridge LNVPS named, not one the daemon - /// assumes: a node told to use a different bridge would otherwise report on - /// an interface that does not exist and look broken. + /// The status report is servable on a machine with no data plane at all — + /// the state a node is in before it has been configured, and one the health + /// gate has to be able to read rather than time out on. #[tokio::test] - async fn the_status_observes_the_bridge_lnvps_named() { + async fn the_status_is_served_before_there_is_a_data_plane() { let keys = Keys::generate(); let addr: SocketAddr = ADDR.parse().unwrap(); - let state = ControlState::with_bridge(keys.public_key(), addr, "br-other"); - assert_eq!(state.bridge, "br-other"); - assert_eq!( - ControlState::new(keys.public_key(), addr).bridge, - "br-lnvps" + let state = ControlState::new( + keys.public_key(), + addr, + Arc::new(crate::net::tests::FakeKernel::default()), ); - // And the report is servable on a machine with no data plane at all, - // which is exactly the state a node is in before it is configured. let url = format!("https://{ADDR}/api/v1/status"); let auth = auth_header(&keys, &url, "GET", b""); let code = status_of(Arc::new(state), get("/api/v1/status", Some(&auth))).await; diff --git a/lnvps_node/src/lib.rs b/lnvps_node/src/lib.rs index e5a71472..94b0e074 100644 --- a/lnvps_node/src/lib.rs +++ b/lnvps_node/src/lib.rs @@ -14,7 +14,9 @@ //! - [`inventory`] — what the node reports about the machine. //! - [`api`] — outbound calls to LNVPS, the only direction that works before //! there is a tunnel. -//! - [`net`] — applying the data plane LNVPS asked for, with `ip` and `wg`. +//! - [`net`] — applying the data plane LNVPS asked for, over netlink. +//! - [`netns`] — the namespace that data plane lives in, so LNVPS configures +//! its own network rather than the operator's. //! - [`wgkey`] — the node's WireGuard key, generated here and never sent. //! - [`config`] — configuration, including where the control API may listen. @@ -25,5 +27,6 @@ pub mod control_auth; pub mod credential; pub mod inventory; pub mod net; +pub mod netns; pub mod tls; pub mod wgkey; diff --git a/lnvps_node/src/main.rs b/lnvps_node/src/main.rs index 8bbd9749..e18f0d7b 100644 --- a/lnvps_node/src/main.rs +++ b/lnvps_node/src/main.rs @@ -119,10 +119,7 @@ async fn dataplane(config_path: &Path, action: DataplaneAction) -> Result<()> { // have?" is the question an operator asks when something is wrong, and it // must not fail because the token is missing or LNVPS is unreachable. if let DataplaneAction::Observe = action { - let state = lnvps_node::net::observe( - &lnvps_node::net::SystemCommands, - lnvps_node::net::DEFAULT_BRIDGE, - )?; + let state = lnvps_node::net::observe(&lnvps_node::net::Kernel::new()?).await?; println!("{}", serde_json::to_string_pretty(&state)?); return Ok(()); } @@ -143,12 +140,8 @@ async fn dataplane(config_path: &Path, action: DataplaneAction) -> Result<()> { match action { DataplaneAction::Show => println!("{}", serde_json::to_string_pretty(&desired)?), DataplaneAction::Apply => { - let applied = lnvps_node::net::apply( - &lnvps_node::net::SystemCommands, - &desired, - &key, - &config.state_dir, - )?; + let kernel = lnvps_node::net::Kernel::new()?; + let applied = lnvps_node::net::apply(&kernel, &desired, &key).await?; for line in applied { println!("{line}"); } @@ -179,23 +172,24 @@ async fn run(config_path: &Path) -> Result<()> { // the address being checked for is one this brings into existence: the // control API binds the tunnel interface, and on a fresh machine the tunnel // does not exist until now. - let bridge = match apply_dataplane(&config).await { - Ok(bridge) => bridge, + let kernel = Arc::new(lnvps_node::net::Kernel::new()?); + if let Err(e) = apply_dataplane(&config, kernel.as_ref()).await { // Not fatal. A node whose tunnel is already up from a previous run must // keep serving through an LNVPS outage — refusing to start would turn // an API blip into every node on the platform going dark. - Err(e) => { - log::warn!( - "Could not apply the data plane ({e}); continuing with whatever this machine \ - already has configured" - ); - lnvps_node::net::DEFAULT_BRIDGE.to_string() - } - }; + log::warn!( + "Could not apply the data plane ({e}); continuing with whatever this machine \ + already has configured" + ); + } // Decision 13: the address must belong to the tunnel interface, checked - // against the interface itself. - let addrs = config::interface_addresses(&control.tunnel_interface)?; + // against the interface itself — inside the data plane namespace, which is + // the only place that interface exists. + let interface = control.tunnel_interface.clone(); + let addrs = kernel + .namespace() + .enter(move || config::interface_addresses(&interface))?; config::validate_listen_address(control.listen, &addrs)?; let tls = tls::load_or_generate(&config.state_dir, Some(control.listen))?; @@ -214,26 +208,35 @@ async fn run(config_path: &Path) -> Result<()> { // end: guests come and go, and a node that only configured itself at // startup would route a departed customer's address until it was restarted. let refresh = config.clone(); + let refresh_kernel = kernel.clone(); tokio::spawn(async move { let interval = Duration::from_secs(refresh.heartbeat_secs.max(10)); loop { tokio::time::sleep(interval).await; - if let Err(e) = apply_dataplane(&refresh).await { + if let Err(e) = apply_dataplane(&refresh, refresh_kernel.as_ref()).await { log::warn!("Data plane refresh failed: {e}"); } } }); - control::serve( - Arc::new(ControlState::with_bridge(control_pubkey, addr, &bridge)), - addr, + // Bound inside the namespace, because that is where the tunnel address + // lives. The socket keeps that namespace while the rest of the daemon stays + // in the machine's own, which is what lets it go on reaching LNVPS. + let listener = kernel.namespace().enter(move || { + std::net::TcpListener::bind(addr) + .with_context(|| format!("Cannot bind the control API to {addr}")) + })?; + + control::serve_on( + Arc::new(ControlState::new(control_pubkey, addr, kernel)), + listener, tls, ) .await } -/// Fetch the data plane and apply it, returning the bridge LNVPS named. -async fn apply_dataplane(config: &NodeConfig) -> Result { +/// Fetch the data plane and apply it. +async fn apply_dataplane(config: &NodeConfig, kernel: &dyn lnvps_node::net::NetOps) -> Result<()> { let credential = Credential::load_checked(&config.credential)?; let api = lnvps_node::api::LnvpsApi::new(&config.api_url, &credential)?; @@ -247,14 +250,9 @@ async fn apply_dataplane(config: &NodeConfig) -> Result { api.request_tunnel(&key.public_bytes()).await?; let desired = api.dataplane().await?; - let applied = lnvps_node::net::apply( - &lnvps_node::net::SystemCommands, - &desired, - &key, - &config.state_dir, - )?; + let applied = lnvps_node::net::apply(kernel, &desired, &key).await?; if !applied.is_empty() { log::debug!("Applied data plane: {}", applied.join("; ")); } - Ok(desired.bridge) + Ok(()) } diff --git a/lnvps_node/src/net.rs b/lnvps_node/src/net.rs index 0b90a37b..d70be8e3 100644 --- a/lnvps_node/src/net.rs +++ b/lnvps_node/src/net.rs @@ -1,25 +1,35 @@ //! Applying the data plane LNVPS asked for. //! -//! The daemon configures the machine itself, with `ip` and `wg`, rather than -//! writing files for something else to read. A marketplace node runs on -//! hardware LNVPS does not own: a data plane that depends on the operator -//! having wired it up correctly is one whose mistakes surface as a customer's -//! VM having no network. Applying it here means it re-converges on every -//! refresh instead. +//! The daemon configures the machine itself rather than writing files for +//! something else to read. A marketplace node runs on hardware LNVPS does not +//! own: a data plane that depends on the operator having wired it up correctly +//! is one whose mistakes surface as a customer's VM having no network. //! -//! Everything is idempotent and stated declaratively — `ip addr replace`, `ip -//! route replace`, `wg set` — so a node that is already correct is not -//! disturbed, and a node that has drifted is corrected without being torn down. +//! Interfaces, addresses and routes are managed over **netlink**, not by +//! shelling out to `ip`. Netlink is the interface the kernel actually offers; +//! `ip` is a program that formats netlink messages and then formats the answer +//! back into text for us to parse. Going direct means no dependency on +//! iproute2's presence or version, no output parsing that changes between +//! releases, no arguments to quote, and errors that arrive as kernel error +//! codes instead of a line of English on stderr. //! -//! Commands go through [`CommandRunner`] because they run as root on somebody -//! else's machine: the exact command issued is the thing worth asserting, which -//! needs the process boundary replaced rather than mocked around. +//! Everything is stated declaratively and converges: a node that is already +//! right is not disturbed, and one that has drifted is corrected without being +//! torn down. +//! +//! The kernel calls sit behind [`NetOps`] so the orchestration can be tested +//! without root: what is worth asserting is *what the node decides to do*. +//! Whether the netlink implementation of those decisions really works on a +//! kernel is proven by the netns end-to-end harness, which runs both ends of a +//! real tunnel and pings across it. use std::collections::HashSet; +use std::net::IpAddr; use std::path::Path; -use std::process::Command; use anyhow::{Context, Result, bail}; +use async_trait::async_trait; +use ipnetwork::IpNetwork; use serde::{Deserialize, Serialize}; use crate::wgkey::{self, NodeKey}; @@ -27,39 +37,15 @@ use crate::wgkey::{self, NodeKey}; /// The tunnel interface the node terminates its data plane on. pub const TUNNEL_INTERFACE: &str = "wg0"; -/// The bridge guests sit on when LNVPS has not said otherwise — which is only -/// before the first data plane has been fetched. -pub const DEFAULT_BRIDGE: &str = "br-lnvps"; - -/// Runs a command and reports what happened. -pub trait CommandRunner: Send + Sync { - /// Run `program` with `args`, returning `(exit ok, stdout)`. - /// - /// Failure to *launch* is an error; a non-zero exit is not, because several - /// callers here use a failing command as a question ("does this interface - /// exist?") rather than as a fault. - fn run(&self, program: &str, args: &[&str]) -> Result<(bool, String)>; -} - -/// Runs commands as real processes. -pub struct SystemCommands; - -impl CommandRunner for SystemCommands { - fn run(&self, program: &str, args: &[&str]) -> Result<(bool, String)> { - let out = Command::new(program) - .args(args) - .output() - .with_context(|| format!("Cannot run {program}: is it installed?"))?; - let text = if out.status.success() { - String::from_utf8_lossy(&out.stdout).to_string() - } else { - // The failure text, not the empty stdout: a caller reporting why - // something did not apply needs what the tool said. - String::from_utf8_lossy(&out.stderr).to_string() - }; - Ok((out.status.success(), text)) - } -} +/// The bridge guests sit on. +/// +/// A constant here rather than a field in the data-plane document, because the +/// daemon needs the name before it has ever spoken to LNVPS: `dataplane +/// observe` reports on it without a credential, and an operator debugging a +/// node asks about it offline. A document that could name a different bridge +/// would leave the node holding two answers to one question. LNVPS holds the +/// same constant, and the end-to-end harness asserts the two agree. +pub const GUEST_BRIDGE: &str = "br-lnvps"; /// The desired data plane, as LNVPS states it. /// @@ -69,7 +55,6 @@ impl CommandRunner for SystemCommands { #[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] pub struct DesiredDataPlane { pub tunnel: DesiredTunnel, - pub bridge: String, /// Gateway addresses this node answers for on the bridge. #[serde(default)] pub gateways: Vec, @@ -97,6 +82,64 @@ pub struct DesiredGuest { pub mac: Option, } +/// How a WireGuard interface should be configured. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WgSettings { + /// Base64, as WireGuard states keys. + pub private_key: String, + pub peer_public_key: String, + pub endpoint: String, + pub keepalive: Option, + /// Everything, for a node: its guests use LNVPS addresses, so no traffic of + /// theirs belongs anywhere else. + pub allowed_ips: Vec, +} + +/// What a WireGuard interface currently is. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct WgObserved { + /// Peers keyed by public key, with seconds since each last handshake. + pub peers: Vec<(String, Option)>, +} + +/// The kernel operations the data plane needs. +/// +/// A trait so the orchestration above it can be tested without root, and so the +/// end-to-end harness can drive the same code inside a network namespace. +#[async_trait] +pub trait NetOps: Send + Sync { + /// Whether a link exists. + async fn link_exists(&self, name: &str) -> Result; + /// Create a WireGuard interface. + async fn create_wireguard(&self, name: &str) -> Result<()>; + /// Create a bridge. + async fn create_bridge(&self, name: &str) -> Result<()>; + /// Bring a link up with the given MTU. + async fn set_up(&self, name: &str, mtu: u32) -> Result<()>; + /// Whether a link is up, and its MTU. + async fn link_state(&self, name: &str) -> Result<(bool, Option)>; + + async fn addresses(&self, name: &str) -> Result>; + async fn add_address(&self, name: &str, address: IpNetwork) -> Result<()>; + async fn del_address(&self, name: &str, address: IpNetwork) -> Result<()>; + + /// Destinations routed out of `name`. + async fn routes(&self, name: &str) -> Result>; + async fn add_route(&self, destination: IpNetwork, name: &str) -> Result<()>; + async fn del_route(&self, destination: IpNetwork, name: &str) -> Result<()>; + + /// Configure the WireGuard interface: key, peer, allowed IPs. + async fn configure_wireguard(&self, name: &str, settings: &WgSettings) -> Result<()>; + /// Remove a peer by public key. + async fn remove_wireguard_peer(&self, name: &str, public_key: &str) -> Result<()>; + /// Read the interface back, or `None` when it does not exist. + async fn wireguard_state(&self, name: &str) -> Result>; + + /// Read a kernel knob, or `None` when this kernel does not have it. + async fn sysctl(&self, key: &str) -> Result>; + async fn set_sysctl(&self, key: &str, value: &str) -> Result<()>; +} + /// What the machine currently looks like. /// /// Reported to LNVPS over the control API, where it is the first thing the @@ -105,7 +148,6 @@ pub struct DesiredGuest { /// the two disagree. #[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] pub struct DataPlaneState { - /// Whether `wg0` exists and is up. pub tunnel_up: bool, /// Seconds since the last handshake with the route server. `None` when /// there has never been one, which is the difference between "configured" @@ -132,712 +174,833 @@ impl DataPlaneState { /// Apply `desired` to this machine. /// -/// Returns the commands that were actually run, so `dataplane apply` can show -/// an operator what changed and a test can assert it. -pub fn apply( - runner: &dyn CommandRunner, +/// Returns a description of what was changed, so `dataplane apply` can show an +/// operator what happened and a test can assert it. An empty list means the +/// machine was already right, which is the normal case on every refresh after +/// the first. +pub async fn apply( + ops: &dyn NetOps, desired: &DesiredDataPlane, key: &NodeKey, - state_dir: &Path, ) -> Result> { - let mut applied = Vec::new(); - apply_tunnel(runner, desired, key, state_dir, &mut applied)?; - apply_bridge(runner, desired, &mut applied)?; - apply_forwarding(runner, &mut applied)?; - Ok(applied) + let mut changed = Vec::new(); + apply_tunnel(ops, desired, key, &mut changed).await?; + apply_bridge(ops, desired, &mut changed).await?; + apply_forwarding(ops, &mut changed).await?; + Ok(changed) } /// Bring up `wg0` and point the default route down it. -fn apply_tunnel( - runner: &dyn CommandRunner, +async fn apply_tunnel( + ops: &dyn NetOps, desired: &DesiredDataPlane, key: &NodeKey, - state_dir: &Path, - applied: &mut Vec, + changed: &mut Vec, ) -> Result<()> { - let mtu = desired.tunnel.mtu.to_string(); - if !link_exists(runner, TUNNEL_INTERFACE)? { - run( - runner, - "ip", - &["link", "add", TUNNEL_INTERFACE, "type", "wireguard"], - applied, - )?; - } - - // The private key is handed over as a path. An argument would be visible in - // `ps` to every user on the machine, and a marketplace node usually has - // more than one login. - let key_file = wgkey::write_private_key_file(state_dir, key)?; - let key_file = key_file.to_string_lossy().to_string(); - run( - runner, - "wg", - &["set", TUNNEL_INTERFACE, "private-key", &key_file], - applied, - )?; - - let server_key = wgkey::parse_public_key(&desired.tunnel.server_public_key)?; - let keepalive = desired.tunnel.keepalive.unwrap_or(0).to_string(); - let mut peer: Vec<&str> = vec![ - "set", - TUNNEL_INTERFACE, - "peer", - &server_key, - "endpoint", - &desired.tunnel.endpoint, + if !ops.link_exists(TUNNEL_INTERFACE).await? { + ops.create_wireguard(TUNNEL_INTERFACE).await?; + changed.push(format!("created {TUNNEL_INTERFACE}")); + } + + let peer_key = wgkey::parse_public_key(&desired.tunnel.server_public_key)?; + let settings = WgSettings { + private_key: key.private_base64(), + peer_public_key: peer_key.clone(), + endpoint: desired.tunnel.endpoint.clone(), + keepalive: desired.tunnel.keepalive, // Everything goes up the tunnel: the node's guests use LNVPS addresses, // so there is no traffic of theirs that belongs anywhere else. - "allowed-ips", - "0.0.0.0/0,::/0", - ]; - if desired.tunnel.keepalive.is_some() { - peer.extend_from_slice(&["persistent-keepalive", &keepalive]); - } - run(runner, "wg", &peer, applied)?; + allowed_ips: vec![ + "0.0.0.0/0".parse().expect("a constant CIDR"), + "::/0".parse().expect("a constant CIDR"), + ], + }; + ops.configure_wireguard(TUNNEL_INTERFACE, &settings).await?; + changed.push(format!("configured {TUNNEL_INTERFACE}")); // A peer that is not the route server has no business on this interface. // It would most likely be a stale key from a re-key, still able to send // traffic that the node treats as coming from LNVPS. - for stale in stale_peers(runner, &server_key)? { - run( - runner, - "wg", - &["set", TUNNEL_INTERFACE, "peer", &stale, "remove"], - applied, - )?; + if let Some(observed) = ops.wireguard_state(TUNNEL_INTERFACE).await? { + for (stale, _) in observed.peers.iter().filter(|(k, _)| *k != peer_key) { + ops.remove_wireguard_peer(TUNNEL_INTERFACE, stale).await?; + changed.push(format!("removed stale peer {stale}")); + } } - for address in [&desired.tunnel.address4, &desired.tunnel.address6] - .into_iter() - .flatten() - { - run( - runner, - "ip", - &["addr", "replace", address, "dev", TUNNEL_INTERFACE], - applied, - )?; - } + let want = tunnel_addresses(desired)?; + sync_addresses(ops, TUNNEL_INTERFACE, &want, changed).await?; // Not 1500: WireGuard's overhead comes off it, and guessing wrong hangs // large transfers rather than failing outright. - run( - runner, - "ip", - &["link", "set", TUNNEL_INTERFACE, "mtu", &mtu, "up"], - applied, - )?; - - // No `via`: the tunnel is point-to-point, so the interface names the next - // hop by itself, and naming a gateway would be a second copy of the route - // server's address free to disagree with the one on the peer. - if desired.tunnel.address4.is_some() { - run( - runner, - "ip", - &["route", "replace", "default", "dev", TUNNEL_INTERFACE], - applied, - )?; - } - if desired.tunnel.address6.is_some() { - run( - runner, - "ip", - &["-6", "route", "replace", "default", "dev", TUNNEL_INTERFACE], - applied, - )?; + ops.set_up(TUNNEL_INTERFACE, desired.tunnel.mtu as u32) + .await?; + + // No gateway: the tunnel is point-to-point, so the interface names the next + // hop by itself, and naming one would be a second copy of the route + // server's address free to disagree with the peer's. + let existing = ops.routes(TUNNEL_INTERFACE).await?; + for default in default_routes(desired) { + if !existing.contains(&default) { + ops.add_route(default, TUNNEL_INTERFACE).await?; + changed.push(format!("routed {default} via {TUNNEL_INTERFACE}")); + } } Ok(()) } /// Bring up the guest bridge and route each guest to it. -fn apply_bridge( - runner: &dyn CommandRunner, +async fn apply_bridge( + ops: &dyn NetOps, desired: &DesiredDataPlane, - applied: &mut Vec, + changed: &mut Vec, ) -> Result<()> { - let bridge = desired.bridge.as_str(); - if bridge.is_empty() { - bail!("LNVPS did not name a bridge, so there is nothing to put guests on"); - } - if !link_exists(runner, bridge)? { - run( - runner, - "ip", - &["link", "add", bridge, "type", "bridge"], - applied, - )?; + if !ops.link_exists(GUEST_BRIDGE).await? { + ops.create_bridge(GUEST_BRIDGE).await?; + changed.push(format!("created {GUEST_BRIDGE}")); } // The bridge carries the same payload as the tunnel, so it takes the same // MTU: a guest that sends 1500 bytes into a 1420-byte tunnel produces a // connection that opens and then hangs on the first large transfer. - let mtu = desired.tunnel.mtu.to_string(); - run( - runner, - "ip", - &["link", "set", bridge, "mtu", &mtu, "up"], - applied, - )?; + ops.set_up(GUEST_BRIDGE, desired.tunnel.mtu as u32).await?; // The gateway belongs to the range, not to this node, and the guest // believes it is on-link. Held as a host address so the node answers for it // without claiming the rest of the range is local — the other addresses in // it live on other nodes, up the tunnel. + let mut want = Vec::new(); for gateway in &desired.gateways { - let addr = host_prefix(gateway)?; - run( - runner, - "ip", - &["addr", "replace", &addr, "dev", bridge], - applied, - )?; + want.push(host_prefix(gateway)?); } + sync_addresses(ops, GUEST_BRIDGE, &want, changed).await?; // The guest thinks its neighbours are on-link and will ARP for them; proxy // ARP is what lets the node answer and pull that traffic up the tunnel // instead of it disappearing into a link that has no such address. for knob in [ - "net.ipv4.conf.NAME.proxy_arp=1", - "net.ipv6.conf.NAME.proxy_ndp=1", + format!("net/ipv4/conf/{GUEST_BRIDGE}/proxy_arp"), + format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp"), ] { - // Interface names appear in sysctl keys with dots replaced, or the key - // itself becomes ambiguous. - let setting = knob.replace("NAME", &bridge.replace('.', "/")); - run(runner, "sysctl", &["-w", &setting], applied)?; + set_if_needed(ops, &knob, "1", changed).await?; } - // What belongs on this bridge: the guests, plus the gateways the node - // answers for. Both are kept so the stale sweep below cannot delete the - // bridge's own addressing while tidying up after a departed guest. - let mut want: HashSet = desired.guests.iter().map(|g| g.address.clone()).collect(); - for gateway in &desired.gateways { - want.insert(host_prefix(gateway)?); - } - let want: HashSet = want; - // Sorted, so a node applying the same document twice runs the same - // commands in the same order and a diff of two runs means something. - let mut guests: Vec<&String> = desired.guests.iter().map(|g| &g.address).collect(); - guests.sort(); - for address in guests { - run( - runner, - "ip", - &["route", "replace", address, "dev", bridge], - applied, - )?; + let mut guests: HashSet = HashSet::new(); + for guest in &desired.guests { + guests.insert(host_prefix(&guest.address)?); + } + let existing: HashSet = ops.routes(GUEST_BRIDGE).await?.into_iter().collect(); + + let mut to_add: Vec<&IpNetwork> = guests.difference(&existing).collect(); + to_add.sort(); + for address in to_add { + ops.add_route(*address, GUEST_BRIDGE).await?; + changed.push(format!("routed {address} to {GUEST_BRIDGE}")); } + // A guest that has been deleted or moved must stop being routed here at // once: its address goes back in the pool and may already be somebody - // else's. - for stale in stale_routes(runner, bridge, &want)? { - run( - runner, - "ip", - &["route", "del", &stale, "dev", bridge], - applied, - )?; + // else's. Routes the kernel maintains for the link itself are left alone — + // deleting the IPv6 link-local prefix to tidy a list would take the + // interface's own connectivity with it. + let mut to_drop: Vec<&IpNetwork> = existing + .difference(&guests) + .filter(|r| !is_link_local(r)) + .collect(); + to_drop.sort(); + for address in to_drop { + ops.del_route(*address, GUEST_BRIDGE).await?; + changed.push(format!("unrouted {address} from {GUEST_BRIDGE}")); } Ok(()) } /// A node that does not forward is a node whose guests have no network at all. -fn apply_forwarding(runner: &dyn CommandRunner, applied: &mut Vec) -> Result<()> { - for setting in ["net.ipv4.ip_forward=1", "net.ipv6.conf.all.forwarding=1"] { - run(runner, "sysctl", &["-w", setting], applied)?; +async fn apply_forwarding(ops: &dyn NetOps, changed: &mut Vec) -> Result<()> { + for knob in ["net/ipv4/ip_forward", "net/ipv6/conf/all/forwarding"] { + set_if_needed(ops, knob, "1", changed).await?; + } + Ok(()) +} + +/// Make the addresses on `name` exactly `want`. +async fn sync_addresses( + ops: &dyn NetOps, + name: &str, + want: &[IpNetwork], + changed: &mut Vec, +) -> Result<()> { + let existing = ops.addresses(name).await?; + for address in want { + if !existing.contains(address) { + ops.add_address(name, *address).await?; + changed.push(format!("added {address} to {name}")); + } + } + for address in &existing { + // A link-local address is the kernel's, not ours. Removing it would + // remove the interface's ability to talk to itself, on every refresh. + if is_link_local(address) || want.contains(address) { + continue; + } + ops.del_address(name, *address).await?; + changed.push(format!("removed {address} from {name}")); } Ok(()) } +/// Write a kernel knob only when it does not already say what it should. +/// +/// A knob the kernel does not have is skipped rather than fatal: IPv6 can be +/// compiled out, and a node with no IPv6 guests is still a working node. +async fn set_if_needed( + ops: &dyn NetOps, + key: &str, + value: &str, + changed: &mut Vec, +) -> Result<()> { + match ops.sysctl(key).await? { + Some(current) if current.trim() == value => Ok(()), + Some(_) => { + ops.set_sysctl(key, value).await?; + changed.push(format!("set {key}={value}")); + Ok(()) + } + None => Ok(()), + } +} + +/// The addresses the tunnel interface should carry. +fn tunnel_addresses(desired: &DesiredDataPlane) -> Result> { + [&desired.tunnel.address4, &desired.tunnel.address6] + .into_iter() + .flatten() + .map(|a| { + a.parse::() + .with_context(|| format!("LNVPS sent {a}, which is not an address")) + }) + .collect() +} + +/// The default routes to install, one per family the tunnel has an address in. +/// +/// A family with no address gets no default route: it would black-hole that +/// family's traffic rather than leaving the machine's own routing to handle it. +fn default_routes(desired: &DesiredDataPlane) -> Vec { + let mut out = Vec::new(); + if desired.tunnel.address4.is_some() { + out.push("0.0.0.0/0".parse().expect("a constant CIDR")); + } + if desired.tunnel.address6.is_some() { + out.push("::/0".parse().expect("a constant CIDR")); + } + out +} + +/// `203.0.113.1` -> `203.0.113.1/32`, and the v6 equivalent. A value that +/// already carries a prefix is taken as it is. +fn host_prefix(address: &str) -> Result { + if address.contains('/') { + return address + .parse::() + .with_context(|| format!("{address} is not a CIDR")); + } + let ip: IpAddr = address + .parse() + .with_context(|| format!("{address} is not an IP address"))?; + Ok(IpNetwork::from(ip)) +} + +/// Addresses the kernel manages for itself. +fn is_link_local(address: &IpNetwork) -> bool { + match address.ip() { + IpAddr::V4(v4) => v4.is_link_local() || v4.is_loopback(), + IpAddr::V6(v6) => (v6.segments()[0] & 0xffc0) == 0xfe80 || v6.is_loopback(), + } +} + /// Read back what the machine actually has. -pub fn observe(runner: &dyn CommandRunner, bridge: &str) -> Result { - let (tunnel_up, tunnel_mtu) = link_state(runner, TUNNEL_INTERFACE)?; - let (bridge_up, _) = link_state(runner, bridge)?; +pub async fn observe(ops: &dyn NetOps) -> Result { + let (tunnel_up, tunnel_mtu) = ops.link_state(TUNNEL_INTERFACE).await?; + let (bridge_up, _) = ops.link_state(GUEST_BRIDGE).await?; + let last_handshake_secs = ops + .wireguard_state(TUNNEL_INTERFACE) + .await? + .and_then(|w| w.peers.into_iter().filter_map(|(_, age)| age).min()); Ok(DataPlaneState { tunnel_up, tunnel_mtu, - last_handshake_secs: last_handshake(runner)?, + last_handshake_secs, bridge_up, - forwarding4: sysctl_enabled(runner, "net.ipv4.ip_forward")?, - forwarding6: sysctl_enabled(runner, "net.ipv6.conf.all.forwarding")?, - routed_guests: routed_addresses(runner, bridge)?.len(), + forwarding4: enabled(ops, "net/ipv4/ip_forward").await?, + forwarding6: enabled(ops, "net/ipv6/conf/all/forwarding").await?, + routed_guests: ops.routes(GUEST_BRIDGE).await?.len(), }) } -/// Whether a link exists at all. -fn link_exists(runner: &dyn CommandRunner, name: &str) -> Result { - Ok(runner.run("ip", &["link", "show", name])?.0) +async fn enabled(ops: &dyn NetOps, key: &str) -> Result { + Ok(ops + .sysctl(key) + .await? + .map(|v| v.trim() == "1") + .unwrap_or(false)) } -/// Whether a link is up, and its MTU. -fn link_state(runner: &dyn CommandRunner, name: &str) -> Result<(bool, Option)> { - let (ok, out) = runner.run("ip", &["-j", "link", "show", name])?; - if !ok { - return Ok((false, None)); +/// Where the kernel exposes its knobs. A path rather than the `sysctl` binary: +/// one less program a node has to have installed, and a write that either +/// happens or reports why. +const PROC_SYS: &str = "/proc/sys"; + +/// Read a kernel knob from `/proc/sys`. +pub fn read_sysctl(root: &Path, key: &str) -> Result> { + let path = root.join(key); + match std::fs::read_to_string(&path) { + Ok(value) => Ok(Some(value)), + // Absent means this kernel does not have the knob — IPv6 can be + // compiled out — which is a fact about the machine, not a failure. + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e).with_context(|| format!("Cannot read {}", path.display())), } - let links: Vec = serde_json::from_str(&out).unwrap_or_default(); - let Some(link) = links.first() else { - return Ok((false, None)); - }; - // `operstate` rather than the UP flag: an interface can be administratively - // up with no carrier, and for a tunnel that is exactly the broken case. - let up = link - .get("flags") - .and_then(|f| f.as_array()) - .map(|f| f.iter().any(|v| v.as_str() == Some("UP"))) - .unwrap_or(false); - let mtu = link.get("mtu").and_then(|m| m.as_u64()).map(|m| m as u32); - Ok((up, mtu)) } -/// Seconds since the route server last completed a handshake. -fn last_handshake(runner: &dyn CommandRunner) -> Result> { - let (ok, out) = runner.run("wg", &["show", TUNNEL_INTERFACE, "latest-handshakes"])?; - if !ok { - return Ok(None); - } - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map(|d| d.as_secs()) - .unwrap_or(0); - let latest = out - .lines() - .filter_map(|line| line.split_whitespace().nth(1)) - .filter_map(|t| t.parse::().ok()) - // Zero means "never", not "in 1970"; reporting an age of half a century - // would look like a stale tunnel rather than one that has never worked. - .filter(|t| *t > 0) - .max(); - Ok(latest.map(|t| now.saturating_sub(t))) +/// Write a kernel knob to `/proc/sys`. +pub fn write_sysctl(root: &Path, key: &str, value: &str) -> Result<()> { + let path = root.join(key); + std::fs::write(&path, value).with_context(|| format!("Cannot set {}", path.display())) } -/// Whether a sysctl is on. -fn sysctl_enabled(runner: &dyn CommandRunner, name: &str) -> Result { - let (ok, out) = runner.run("sysctl", &["-n", name])?; - Ok(ok && out.trim() == "1") -} +pub use kernel::Kernel; -/// Peers configured on the tunnel that are not the route server. -fn stale_peers(runner: &dyn CommandRunner, server_key: &str) -> Result> { - let (ok, out) = runner.run("wg", &["show", TUNNEL_INTERFACE, "peers"])?; - if !ok { - return Ok(vec![]); - } - Ok(out - .lines() - .map(str::trim) - .filter(|l| !l.is_empty() && *l != server_key) - .map(str::to_string) - .collect()) +/// A machine whose network cannot be read. +/// +/// Not a mock: it is what a node that has never been configured looks like from +/// outside, and reporting that truthfully is what lets the health gate say +/// "this node has no data plane" instead of timing out. +pub struct UnavailableKernel; + +#[async_trait] +impl NetOps for UnavailableKernel { + async fn link_exists(&self, _name: &str) -> Result { + Ok(false) + } + async fn create_wireguard(&self, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn create_bridge(&self, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn set_up(&self, _name: &str, _mtu: u32) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn link_state(&self, _name: &str) -> Result<(bool, Option)> { + Ok((false, None)) + } + async fn addresses(&self, _name: &str) -> Result> { + Ok(vec![]) + } + async fn add_address(&self, _name: &str, _address: IpNetwork) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn del_address(&self, _name: &str, _address: IpNetwork) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn routes(&self, _name: &str) -> Result> { + Ok(vec![]) + } + async fn add_route(&self, _destination: IpNetwork, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn del_route(&self, _destination: IpNetwork, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn configure_wireguard(&self, _name: &str, _settings: &WgSettings) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn remove_wireguard_peer(&self, _name: &str, _public_key: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn wireguard_state(&self, _name: &str) -> Result> { + Ok(None) + } + async fn sysctl(&self, _key: &str) -> Result> { + Ok(None) + } + async fn set_sysctl(&self, _key: &str, _value: &str) -> Result<()> { + bail!("This node cannot configure its network") + } } -/// Addresses currently routed to the bridge. -fn routed_addresses(runner: &dyn CommandRunner, bridge: &str) -> Result> { - let mut out = Vec::new(); - // Both families asked for separately: `ip route show` is IPv4 only, so a v6 - // guest would look unrouted on every pass and be re-added forever. - for family in ["-4", "-6"] { - let (ok, text) = runner.run("ip", &[family, "-j", "route", "show", "dev", bridge])?; - if !ok { - continue; +/// The real implementation: netlink for links, addresses and routes, the +/// kernel's WireGuard netlink interface for the tunnel, and `/proc/sys` for the +/// forwarding knobs. +mod kernel { + use super::*; + + use defguard_wireguard_rs::key::Key; + use defguard_wireguard_rs::net::IpAddrMask; + use defguard_wireguard_rs::peer::Peer; + use defguard_wireguard_rs::{ + InterfaceConfiguration, Kernel as WgKernel, WGApi, WireguardInterfaceApi, + }; + use futures_util::TryStreamExt; + use netlink_packet_route::address::AddressAttribute; + use netlink_packet_route::link::LinkFlags; + use netlink_packet_route::route::{RouteAddress, RouteAttribute, RouteHeader}; + use rtnetlink::{Handle, LinkBridge, LinkUnspec, RouteMessageBuilder, new_connection}; + + /// Talks to the kernel, inside the data plane's own network namespace. + /// + /// The namespace is why this type exists at all: a marketplace node is + /// often not only a marketplace node, and configuring routes, forwarding + /// and proxy ARP in the machine's own namespace would be configuring the + /// operator's network for them. See [`crate::netns`]. + pub struct Kernel { + handle: Handle, + namespace: crate::netns::Handle, + } + + impl Kernel { + /// Open a netlink connection inside the data plane namespace, + /// creating the namespace if this is the first run. + pub fn new() -> Result { + Self::in_namespace(crate::netns::ensure_default()?) } - let routes: Vec = serde_json::from_str(&text).unwrap_or_default(); - for route in routes { - let Some(dst) = route.get("dst").and_then(|d| d.as_str()) else { - continue; - }; - if dst == "default" { - continue; + + /// Same, for an already-open namespace. Used by the end-to-end harness, + /// which builds its own. + pub fn in_namespace(namespace: crate::netns::Handle) -> Result { + // The socket is opened *inside* the namespace: a netlink socket + // belongs to the namespace it was created in, so everything sent + // over this one lands there no matter which thread sends it. + // + // The runtime handle goes with it because the socket registers with + // tokio's reactor as it is created, and the thread that enters the + // namespace is a bare one — without this it panics with "there is + // no reactor running", which is a confusing way to discover that a + // namespace and a runtime are different kinds of context. + let runtime = tokio::runtime::Handle::current(); + let (connection, handle, _) = namespace.enter(move || { + let _guard = runtime.enter(); + new_connection().context("Cannot open a netlink socket") + })?; + tokio::spawn(connection); + Ok(Self { handle, namespace }) + } + + /// The namespace this configures. + pub fn namespace(&self) -> &crate::netns::Handle { + &self.namespace + } + + async fn index(&self, name: &str) -> Result> { + let mut links = self + .handle + .link() + .get() + .match_name(name.to_string()) + .execute(); + match links.try_next().await { + Ok(Some(link)) => Ok(Some(link.header.index)), + // "no such device" arrives as an error, and it is an answer + // rather than a fault: the caller is asking whether to create it. + Ok(None) | Err(_) => Ok(None), } - out.push(if dst.contains('/') { - dst.to_string() - } else { - host_prefix(dst)? - }); + } + + async fn require_index(&self, name: &str) -> Result { + self.index(name) + .await? + .with_context(|| format!("Interface {name} does not exist")) } } - Ok(out) -} -/// Routes on the bridge that no guest accounts for. -fn stale_routes( - runner: &dyn CommandRunner, - bridge: &str, - want: &HashSet, -) -> Result> { - Ok(routed_addresses(runner, bridge)? - .into_iter() - .filter(|r| !want.contains(r)) - .collect()) -} + #[async_trait] + impl NetOps for Kernel { + async fn link_exists(&self, name: &str) -> Result { + Ok(self.index(name).await?.is_some()) + } -/// `203.0.113.1` -> `203.0.113.1/32`, and the v6 equivalent. -fn host_prefix(address: &str) -> Result { - if address.contains('/') { - return Ok(address.to_string()); - } - let ip: std::net::IpAddr = address - .parse() - .with_context(|| format!("{address} is not an IP address"))?; - Ok(match ip { - std::net::IpAddr::V4(v4) => format!("{v4}/32"), - std::net::IpAddr::V6(v6) => format!("{v6}/128"), - }) -} + async fn create_wireguard(&self, name: &str) -> Result<()> { + // Deliberately created in the machine's own namespace and then + // moved: a WireGuard interface keeps its UDP socket in the + // namespace it was created in. Created inside, it could only reach + // a route server through itself. This way the encrypted outer + // traffic still leaves by the operator's uplink while everything + // carried over the tunnel stays isolated. + let mut api = WGApi::::new(name.to_string()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + api.create_interface() + .with_context(|| format!("Cannot create WireGuard interface {name}"))?; + self.move_into_namespace(name).await + } -/// Run a command that must succeed, recording it. -fn run( - runner: &dyn CommandRunner, - program: &str, - args: &[&str], - applied: &mut Vec, -) -> Result<()> { - let (ok, out) = runner.run(program, args)?; - let line = format!("{program} {}", args.join(" ")); - if !ok { - bail!("{line}: {}", out.trim()); - } - applied.push(line); - Ok(()) -} + async fn create_bridge(&self, name: &str) -> Result<()> { + // Created inside: nothing about a bridge needs the machine's own + // namespace, and a guest port must never be attachable from there. + self.handle + .link() + .add(LinkBridge::new(name).build()) + .execute() + .await + .with_context(|| format!("Cannot create bridge {name}")) + } -#[cfg(test)] -mod tests { - use super::*; - use std::sync::Mutex; + async fn set_up(&self, name: &str, mtu: u32) -> Result<()> { + let index = self.require_index(name).await?; + self.handle + .link() + .set(LinkUnspec::new_with_index(index).up().mtu(mtu).build()) + .execute() + .await + .with_context(|| format!("Cannot bring {name} up with MTU {mtu}")) + } - /// A machine that answers `ip`/`wg` queries from a fixed script and records - /// everything it was asked to change. - /// - /// The commands run as root on somebody else's hardware, so what is worth - /// asserting is the exact command issued — which needs the process boundary - /// replaced, not mocked around. - struct FakeMachine { - log: Mutex>, - /// Substring -> canned (ok, stdout). - answers: Vec<(&'static str, bool, &'static str)>, - } - - impl FakeMachine { - fn new(answers: Vec<(&'static str, bool, &'static str)>) -> Self { - Self { - log: Mutex::new(Vec::new()), - answers, + async fn link_state(&self, name: &str) -> Result<(bool, Option)> { + let mut links = self + .handle + .link() + .get() + .match_name(name.to_string()) + .execute(); + let Ok(Some(link)) = links.try_next().await else { + return Ok((false, None)); + }; + let up = link.header.flags.contains(LinkFlags::Up); + let mtu = link.attributes.iter().find_map(|a| { + if let netlink_packet_route::link::LinkAttribute::Mtu(mtu) = a { + Some(*mtu) + } else { + None + } + }); + Ok((up, mtu)) + } + + async fn addresses(&self, name: &str) -> Result> { + let Some(index) = self.index(name).await? else { + return Ok(vec![]); + }; + let mut addresses = self + .handle + .address() + .get() + .set_link_index_filter(index) + .execute(); + let mut out = Vec::new(); + while let Some(message) = addresses.try_next().await? { + let prefix = message.header.prefix_len; + for attribute in message.attributes { + if let AddressAttribute::Address(ip) = attribute + && let Ok(network) = IpNetwork::new(ip, prefix) + { + out.push(network); + } + } } + Ok(out) } - fn ran(&self) -> Vec { - self.log.lock().unwrap().clone() + async fn add_address(&self, name: &str, address: IpNetwork) -> Result<()> { + let index = self.require_index(name).await?; + self.handle + .address() + .add(index, address.ip(), address.prefix()) + .execute() + .await + .with_context(|| format!("Cannot add {address} to {name}")) } - } - impl CommandRunner for FakeMachine { - fn run(&self, program: &str, args: &[&str]) -> Result<(bool, String)> { - let line = format!("{program} {}", args.join(" ")); - self.log.lock().unwrap().push(line.clone()); - for (needle, ok, out) in &self.answers { - if line.contains(needle) { - return Ok((*ok, out.to_string())); + async fn del_address(&self, name: &str, address: IpNetwork) -> Result<()> { + let index = self.require_index(name).await?; + let mut addresses = self + .handle + .address() + .get() + .set_link_index_filter(index) + .execute(); + while let Some(message) = addresses.try_next().await? { + let matches = message.header.prefix_len == address.prefix() + && message + .attributes + .iter() + .any(|a| matches!(a, AddressAttribute::Address(ip) if *ip == address.ip())); + if matches { + return self + .handle + .address() + .del(message) + .execute() + .await + .with_context(|| format!("Cannot remove {address} from {name}")); } } - Ok((true, String::new())) + Ok(()) + } + + async fn routes(&self, name: &str) -> Result> { + let Some(index) = self.index(name).await? else { + return Ok(vec![]); + }; + let mut out = Vec::new(); + // Both families asked for separately, because netlink dumps one + // family at a time: a v6 guest would otherwise look unrouted on + // every pass and be re-added forever. + for builder in [ + RouteMessageBuilder::::new().build(), + RouteMessageBuilder::::new().build(), + ] { + let mut routes = self.handle.route().get(builder).execute(); + while let Some(route) = routes.try_next().await? { + let on_this_link = route + .attributes + .iter() + .any(|a| matches!(a, RouteAttribute::Oif(oif) if *oif == index)); + // Only the main table. Giving an interface an address makes + // the kernel write entries into the *local* table for it, + // and treating those as ours means trying to delete the + // bridge's own gateway on every sweep — which fails, and + // rightly so. + if !on_this_link || route.header.table != RouteHeader::RT_TABLE_MAIN { + continue; + } + let destination = route.attributes.iter().find_map(|a| match a { + RouteAttribute::Destination(RouteAddress::Inet(v4)) => { + Some(IpAddr::from(*v4)) + } + RouteAttribute::Destination(RouteAddress::Inet6(v6)) => { + Some(IpAddr::from(*v6)) + } + _ => None, + }); + let prefix = route.header.destination_prefix_length; + let network = match destination { + Some(ip) => IpNetwork::new(ip, prefix).ok(), + // No destination at all is the default route. + None if route.header.address_family + == netlink_packet_route::AddressFamily::Inet => + { + "0.0.0.0/0".parse().ok() + } + None => "::/0".parse().ok(), + }; + if let Some(network) = network { + out.push(network); + } + } + } + Ok(out) + } + + async fn add_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + let index = self.require_index(name).await?; + let message = match destination { + IpNetwork::V4(v4) => RouteMessageBuilder::::new() + .destination_prefix(v4.ip(), v4.prefix()) + .output_interface(index) + .build(), + IpNetwork::V6(v6) => RouteMessageBuilder::::new() + .destination_prefix(v6.ip(), v6.prefix()) + .output_interface(index) + .build(), + }; + self.handle + .route() + .add(message) + .replace() + .execute() + .await + .with_context(|| format!("Cannot route {destination} via {name}")) + } + + async fn del_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + let index = self.require_index(name).await?; + let message = match destination { + IpNetwork::V4(v4) => RouteMessageBuilder::::new() + .destination_prefix(v4.ip(), v4.prefix()) + .output_interface(index) + .build(), + IpNetwork::V6(v6) => RouteMessageBuilder::::new() + .destination_prefix(v6.ip(), v6.prefix()) + .output_interface(index) + .build(), + }; + self.handle + .route() + .del(message) + .execute() + .await + .with_context(|| format!("Cannot remove the route for {destination} on {name}")) + } + + async fn configure_wireguard(&self, name: &str, settings: &WgSettings) -> Result<()> { + // Inside the namespace: the interface was moved there, and + // WireGuard's netlink socket, like every other, belongs to the + // namespace of the thread that opens it. Configuring from outside + // reports "no such device" about an interface that plainly exists. + let (name, settings) = (name.to_string(), settings.clone()); + self.namespace.enter(move || configure(&name, &settings)) } - } - fn desired() -> DesiredDataPlane { - DesiredDataPlane { - tunnel: DesiredTunnel { - address4: Some("10.66.0.2/32".to_string()), - address6: Some("fd00:66::2/128".to_string()), - gateway4: Some("10.66.0.1".to_string()), - gateway6: Some("fd00:66::1".to_string()), - server_public_key: hex::encode([0xab; 32]), - endpoint: "rs1.example:51820".to_string(), - keepalive: Some(25), - mtu: 1420, - }, - bridge: "br-lnvps".to_string(), - gateways: vec!["203.0.113.1".to_string()], - guests: vec![DesiredGuest { - address: "203.0.113.5/32".to_string(), - gateway: "203.0.113.1".to_string(), - mac: Some("aa:bb:cc:dd:ee:ff".to_string()), - }], + async fn remove_wireguard_peer(&self, name: &str, public_key: &str) -> Result<()> { + let (name, public_key) = (name.to_string(), public_key.to_string()); + self.namespace.enter(move || { + let api = WGApi::::new(name.clone()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + let key = Key::from_str(&public_key).context("Not a WireGuard key")?; + api.remove_peer(&key) + .with_context(|| format!("Cannot remove peer {public_key} from {name}")) + }) + } + + async fn wireguard_state(&self, name: &str) -> Result> { + if !self.link_exists(name).await? { + return Ok(None); + } + let name = name.to_string(); + self.namespace + .enter(move || observe_wireguard(&name)) + .map(Some) + } + + async fn sysctl(&self, key: &str) -> Result> { + // `/proc/sys/net` reflects the reading thread's namespace, so this + // has to be read from inside — otherwise the node would report the + // operator's forwarding setting as its own. + let key = key.to_string(); + self.namespace + .enter(move || read_sysctl(Path::new(PROC_SYS), &key)) + } + + async fn set_sysctl(&self, key: &str, value: &str) -> Result<()> { + let (key, value) = (key.to_string(), value.to_string()); + self.namespace + .enter(move || write_sysctl(Path::new(PROC_SYS), &key, &value)) } } - fn key(dir: &Path) -> NodeKey { - wgkey::load_or_generate(dir).unwrap() - } - - /// A node with nothing configured must end up with the whole data plane: - /// tunnel, addresses, MTU, default route, bridge, gateway, guest route and - /// forwarding. Any one of them missing is a customer with no network. - #[test] - fn a_bare_machine_gets_the_whole_data_plane() { - let dir = tempfile::tempdir().unwrap(); - // Neither interface exists yet. - let machine = FakeMachine::new(vec![("ip link show", false, "does not exist")]); - let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); - let script = applied.join("\n"); - - assert!( - script.contains("ip link add wg0 type wireguard"), - "{script}" - ); - assert!(script.contains("wg set wg0 private-key"), "{script}"); - // Everything goes up the tunnel: the guests use LNVPS addresses, so no - // traffic of theirs belongs anywhere else. - assert!(script.contains("allowed-ips 0.0.0.0/0,::/0"), "{script}"); - assert!(script.contains("persistent-keepalive 25"), "{script}"); - assert!( - script.contains("ip addr replace 10.66.0.2/32 dev wg0"), - "{script}" - ); - assert!( - script.contains("ip addr replace fd00:66::2/128 dev wg0"), - "{script}" - ); - assert!(script.contains("ip link set wg0 mtu 1420 up"), "{script}"); - assert!( - script.contains("ip route replace default dev wg0"), - "{script}" - ); - assert!( - script.contains("ip -6 route replace default dev wg0"), - "{script}" - ); - - assert!( - script.contains("ip link add br-lnvps type bridge"), - "{script}" - ); - // The bridge takes the tunnel's MTU: a guest sending 1500 bytes into a - // 1420-byte tunnel opens a connection and then hangs on a large one. - assert!( - script.contains("ip link set br-lnvps mtu 1420 up"), - "{script}" - ); - // The gateway belongs to the range, not the node, and is held as a host - // address so the node answers for it without claiming the rest of the - // range is local. - assert!( - script.contains("ip addr replace 203.0.113.1/32 dev br-lnvps"), - "{script}" - ); - assert!(script.contains("proxy_arp=1"), "{script}"); - assert!(script.contains("proxy_ndp=1"), "{script}"); - assert!( - script.contains("ip route replace 203.0.113.5/32 dev br-lnvps"), - "{script}" - ); - assert!(script.contains("net.ipv4.ip_forward=1"), "{script}"); - assert!( - script.contains("net.ipv6.conf.all.forwarding=1"), - "{script}" - ); - } - - /// The private key reaches `wg` as a path, never as an argument: arguments - /// are visible in `ps` to every user on the machine. - #[test] - fn the_private_key_is_never_an_argument() { - let dir = tempfile::tempdir().unwrap(); - let node_key = key(dir.path()); - let machine = FakeMachine::new(vec![]); - apply(&machine, &desired(), &node_key, dir.path()).unwrap(); - - let script = machine.ran().join("\n"); - assert!( - !script.contains(&node_key.private_base64()), - "the private key was passed on a command line" - ); - assert!(script.contains("private-key"), "{script}"); - } - - /// An interface that already exists is configured, not recreated: - /// recreating it drops the tunnel every time the node refreshes. - #[test] - fn an_existing_interface_is_not_recreated() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![("ip link show", true, "")]); - let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); - let script = applied.join("\n"); - assert!(!script.contains("ip link add"), "{script}"); - assert!(script.contains("ip addr replace"), "{script}"); - } - - /// A peer that is not the route server has no business on this interface — - /// most likely a stale key from a re-key, still able to send traffic the - /// node would treat as LNVPS's. - #[test] - fn a_stale_peer_is_removed() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![("wg show wg0 peers", true, "c3RyYXk=\n")]); - let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); - let script = applied.join("\n"); - assert!( - script.contains("wg set wg0 peer c3RyYXk= remove"), - "{script}" - ); - } - - /// A guest that has been deleted or moved must stop being routed here at - /// once: its address goes back in the pool and may already be somebody - /// else's. The bridge's own gateway must survive that sweep. - #[test] - fn a_departed_guest_stops_being_routed_but_the_gateway_stays() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![( - "-4 -j route show dev br-lnvps", - true, - r#"[{"dst":"203.0.113.5"},{"dst":"203.0.113.9"},{"dst":"203.0.113.1"}]"#, - )]); - let applied = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap(); - let script = applied.join("\n"); - assert!( - script.contains("ip route del 203.0.113.9/32 dev br-lnvps"), - "{script}" - ); - assert!( - !script.contains("ip route del 203.0.113.5/32"), - "a guest that is still here was unrouted" - ); - assert!( - !script.contains("ip route del 203.0.113.1/32"), - "the bridge's own gateway was deleted" - ); - } - - /// A command that fails stops the run and says which one: half a data plane - /// applied silently is worse than none, because it looks configured. - #[test] - fn a_failing_command_is_reported() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![ - ("ip link show", false, ""), - ("ip link add wg0", false, "RTNETLINK answers: not permitted"), - ]); - let err = apply(&machine, &desired(), &key(dir.path()), dir.path()).unwrap_err(); - assert!(format!("{err}").contains("ip link add wg0"), "{err}"); - assert!(format!("{err}").contains("not permitted"), "{err}"); - } - - /// A document naming no bridge would silently configure a tunnel with - /// nothing behind it. - #[test] - fn a_document_without_a_bridge_is_refused() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![]); - let plane = DesiredDataPlane { - bridge: String::new(), - ..desired() + /// Configure a WireGuard interface. Runs on a thread already inside the + /// data plane namespace. + fn configure(name: &str, settings: &WgSettings) -> Result<()> { + let api = WGApi::::new(name.to_string()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + let peer = Peer { + public_key: Key::from_str(&settings.peer_public_key) + .context("The route server's key is not a WireGuard key")?, + endpoint: Some( + resolve(&settings.endpoint) + .with_context(|| format!("Cannot resolve endpoint {}", settings.endpoint))?, + ), + persistent_keepalive_interval: settings.keepalive, + allowed_ips: settings + .allowed_ips + .iter() + .map(|n| IpAddrMask::new(n.ip(), n.prefix())) + .collect(), + ..Default::default() }; - assert!(apply(&machine, &plane, &key(dir.path()), dir.path()).is_err()); - } - - /// A single-stack pool must not produce a default route for the family it - /// has no address in — that route would black-hole traffic instead of - /// letting the machine's own routing handle it. - #[test] - fn a_single_stack_tunnel_only_routes_its_own_family() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![]); - let mut plane = desired(); - plane.tunnel.address6 = None; - let applied = apply(&machine, &plane, &key(dir.path()), dir.path()).unwrap(); - let script = applied.join("\n"); - assert!(script.contains("ip route replace default dev wg0")); - assert!(!script.contains("ip -6 route replace default"), "{script}"); - } - - /// A gateway that is not an address is reported against the value, not as - /// a failing `ip` command: LNVPS sent it, and the node has to say which - /// part of the document it could not use. - #[test] - fn a_gateway_that_is_not_an_address_is_reported() { - let dir = tempfile::tempdir().unwrap(); - let machine = FakeMachine::new(vec![]); - let plane = DesiredDataPlane { - gateways: vec!["not-an-address".to_string()], - ..desired() + // Addresses and MTU are handled through netlink above rather than + // here, so that one code path owns them for both interfaces. + let config = InterfaceConfiguration { + name: name.to_string(), + prvkey: settings.private_key.clone(), + addresses: vec![], + port: 0, + peers: vec![peer], + mtu: None, + fwmark: None, }; - let err = apply(&machine, &plane, &key(dir.path()), dir.path()).unwrap_err(); - assert!(format!("{err:#}").contains("not-an-address"), "{err:#}"); - } - - /// Observation reads the machine rather than remembering what was applied: - /// the point of observing is to catch the case where the two disagree. - #[test] - fn observation_reports_what_the_machine_has() { - let now = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_secs(); - let machine = FakeMachine::new(vec![ - ( - "-j link show wg0", - true, - r#"[{"ifname":"wg0","flags":["POINTOPOINT","NOARP","UP","LOWER_UP"],"mtu":1420}]"#, - ), - ( - "-j link show br-lnvps", - true, - r#"[{"ifname":"br-lnvps","flags":["BROADCAST","MULTICAST","UP"],"mtu":1420}]"#, - ), - ( - "wg show wg0 latest-handshakes", - true, - Box::leak(format!("peerkey\t{}\n", now - 12).into_boxed_str()), - ), - ("sysctl -n", true, "1\n"), - ( - "-4 -j route show dev br-lnvps", - true, - r#"[{"dst":"203.0.113.5"}]"#, - ), - ]); - - let state = observe(&machine, "br-lnvps").unwrap(); - assert!(state.tunnel_up); - assert_eq!(state.tunnel_mtu, Some(1420)); - assert!(state.last_handshake_secs.unwrap() <= 13); - assert!(state.bridge_up); - assert!(state.forwarding4 && state.forwarding6); - assert_eq!(state.routed_guests, 1); - assert!(state.healthy()); - } - - /// `wg0` comes up happily with a peer that never answers, so an interface - /// that has never handshaken is configured, not working — and a node in - /// that state must not be called healthy. - #[test] - fn a_tunnel_that_has_never_handshaken_is_not_healthy() { - let machine = FakeMachine::new(vec![ - ( - "-j link show", - true, - r#"[{"ifname":"wg0","flags":["UP"],"mtu":1420}]"#, - ), - // Zero means never, not 1970: reporting an age of half a century - // would look like a stale tunnel rather than one never used. - ("latest-handshakes", true, "peerkey\t0\n"), - ("sysctl -n", true, "1\n"), - ]); - let state = observe(&machine, "br-lnvps").unwrap(); - assert!(state.tunnel_up); - assert_eq!(state.last_handshake_secs, None); - assert!(!state.healthy()); - } - - /// A machine with nothing configured reports nothing configured, rather - /// than failing: "not set up yet" is a state the gate has to be able to - /// read. - #[test] - fn an_unconfigured_machine_observes_cleanly() { - let machine = FakeMachine::new(vec![("", false, "does not exist")]); - let state = observe(&machine, "br-lnvps").unwrap(); - assert_eq!(state, DataPlaneState::default()); - assert!(!state.healthy()); + api.configure_interface(&config) + .with_context(|| format!("Cannot configure WireGuard interface {name}")) + } + + /// Read a WireGuard interface back. Runs inside the namespace, as above. + fn observe_wireguard(name: &str) -> Result { + let api = WGApi::::new(name.to_string()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + let host = api + .read_interface_data() + .with_context(|| format!("Cannot read WireGuard interface {name}"))?; + let now = std::time::SystemTime::now(); + Ok(WgObserved { + peers: host + .peers + .values() + .map(|p| { + let age = p + .last_handshake + // The zero time means "never", not 1970: reporting + // an age of half a century would look like a stale + // tunnel rather than one that has never worked. + .filter(|t| *t != std::time::UNIX_EPOCH) + .and_then(|t| now.duration_since(t).ok()) + .map(|d| d.as_secs()); + (p.public_key.to_string(), age) + }) + .collect(), + }) + } + + impl Kernel { + /// Move an interface from the machine's namespace into the data + /// plane's. + /// + /// Uses a second netlink socket, opened in the machine's own namespace, + /// because the move is asked of the namespace the interface is *in*. + async fn move_into_namespace(&self, name: &str) -> Result<()> { + let (connection, handle, _) = + new_connection().context("Cannot open a netlink socket")?; + tokio::spawn(connection); + + let mut links = handle.link().get().match_name(name.to_string()).execute(); + let Ok(Some(link)) = links.try_next().await else { + // Already inside: `create_wireguard` is the only caller, and a + // retry after a partial failure finds it where it belongs. + return Ok(()); + }; + handle + .link() + .set( + LinkUnspec::new_with_index(link.header.index) + .setns_by_fd(self.namespace.as_raw_fd()) + .build(), + ) + .execute() + .await + .with_context(|| format!("Cannot move {name} into the data plane namespace")) + } + } + + /// Resolve `host:port`, which is what a node is told to dial. + /// + /// Resolved here rather than passed through as text because the kernel + /// takes an address: a name that does not resolve has to fail with that + /// message, not as a rejected netlink attribute. + fn resolve(endpoint: &str) -> Result { + use std::net::ToSocketAddrs; + endpoint + .to_socket_addrs()? + .next() + .with_context(|| format!("{endpoint} resolved to no addresses")) + } + + trait KeyFromStr: Sized { + fn from_str(value: &str) -> Result; + } + + impl KeyFromStr for Key { + /// WireGuard states keys in base64; the crate wants bytes. + fn from_str(value: &str) -> Result { + use base64::Engine; + let raw = base64::engine::general_purpose::STANDARD + .decode(value.trim()) + .context("A WireGuard key must be base64")?; + let bytes: [u8; 32] = raw + .as_slice() + .try_into() + .map_err(|_| anyhow::anyhow!("A WireGuard key is 32 bytes, got {}", raw.len()))?; + Ok(Key::new(bytes)) + } } } + +#[cfg(test)] +pub mod tests; diff --git a/lnvps_node/src/net/tests.rs b/lnvps_node/src/net/tests.rs new file mode 100644 index 00000000..5b928314 --- /dev/null +++ b/lnvps_node/src/net/tests.rs @@ -0,0 +1,509 @@ +//! Tests for the orchestration: what the node *decides* to do to a machine. +//! +//! The kernel sits behind [`NetOps`], so these run without root and assert the +//! decisions rather than the netlink encoding of them. Whether those decisions +//! really work on a kernel is proven by the end-to-end harness, which builds +//! both ends of a real tunnel in network namespaces and pings across it. + +use std::collections::HashMap; +use std::sync::Mutex; + +use super::*; + +/// A machine that remembers what it was told, and answers questions from what +/// it has been told so far — so a second `apply` sees the first one's work, +/// which is what makes "converges and then goes quiet" testable. +#[derive(Default)] +pub struct FakeKernel { + links: Mutex)>>, + addresses: Mutex>>, + routes: Mutex>>, + wireguard: Mutex>, + settings: Mutex>, + sysctls: Mutex>, + /// Knobs this kernel does not have at all. + missing: Vec, +} + +impl FakeKernel { + fn new() -> Self { + let sysctls = HashMap::from([ + ("net/ipv4/ip_forward".to_string(), "0".to_string()), + ("net/ipv6/conf/all/forwarding".to_string(), "0".to_string()), + ( + format!("net/ipv4/conf/{GUEST_BRIDGE}/proxy_arp"), + "0".to_string(), + ), + ( + format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp"), + "0".to_string(), + ), + ]); + Self { + sysctls: Mutex::new(sysctls), + ..Default::default() + } + } + + fn addresses_of(&self, name: &str) -> Vec { + self.addresses + .lock() + .unwrap() + .get(name) + .cloned() + .unwrap_or_default() + } + + fn routes_of(&self, name: &str) -> Vec { + self.routes + .lock() + .unwrap() + .get(name) + .cloned() + .unwrap_or_default() + } + + fn wg_settings(&self) -> Option { + self.settings.lock().unwrap().clone() + } + + fn sysctl_value(&self, key: &str) -> Option { + self.sysctls.lock().unwrap().get(key).cloned() + } +} + +#[async_trait] +impl NetOps for FakeKernel { + async fn link_exists(&self, name: &str) -> Result { + Ok(self.links.lock().unwrap().contains_key(name)) + } + + async fn create_wireguard(&self, name: &str) -> Result<()> { + self.links + .lock() + .unwrap() + .insert(name.to_string(), (false, None)); + self.wireguard + .lock() + .unwrap() + .insert(name.to_string(), WgObserved::default()); + Ok(()) + } + + async fn create_bridge(&self, name: &str) -> Result<()> { + self.links + .lock() + .unwrap() + .insert(name.to_string(), (false, None)); + Ok(()) + } + + async fn set_up(&self, name: &str, mtu: u32) -> Result<()> { + self.links + .lock() + .unwrap() + .insert(name.to_string(), (true, Some(mtu))); + Ok(()) + } + + async fn link_state(&self, name: &str) -> Result<(bool, Option)> { + Ok(self + .links + .lock() + .unwrap() + .get(name) + .copied() + .unwrap_or((false, None))) + } + + async fn addresses(&self, name: &str) -> Result> { + Ok(self.addresses_of(name)) + } + + async fn add_address(&self, name: &str, address: IpNetwork) -> Result<()> { + self.addresses + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .push(address); + Ok(()) + } + + async fn del_address(&self, name: &str, address: IpNetwork) -> Result<()> { + if let Some(list) = self.addresses.lock().unwrap().get_mut(name) { + list.retain(|a| *a != address); + } + Ok(()) + } + + async fn routes(&self, name: &str) -> Result> { + Ok(self.routes_of(name)) + } + + async fn add_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + self.routes + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .push(destination); + Ok(()) + } + + async fn del_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + if let Some(list) = self.routes.lock().unwrap().get_mut(name) { + list.retain(|r| *r != destination); + } + Ok(()) + } + + async fn configure_wireguard(&self, name: &str, settings: &WgSettings) -> Result<()> { + *self.settings.lock().unwrap() = Some(settings.clone()); + self.wireguard + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .peers + .retain(|(k, _)| *k != settings.peer_public_key); + self.wireguard + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .peers + .push((settings.peer_public_key.clone(), None)); + Ok(()) + } + + async fn remove_wireguard_peer(&self, name: &str, public_key: &str) -> Result<()> { + if let Some(state) = self.wireguard.lock().unwrap().get_mut(name) { + state.peers.retain(|(k, _)| k != public_key); + } + Ok(()) + } + + async fn wireguard_state(&self, name: &str) -> Result> { + Ok(self.wireguard.lock().unwrap().get(name).cloned()) + } + + async fn sysctl(&self, key: &str) -> Result> { + if self.missing.iter().any(|m| m == key) { + return Ok(None); + } + Ok(self.sysctl_value(key)) + } + + async fn set_sysctl(&self, key: &str, value: &str) -> Result<()> { + self.sysctls + .lock() + .unwrap() + .insert(key.to_string(), value.to_string()); + Ok(()) + } +} + +fn desired() -> DesiredDataPlane { + DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some("10.66.0.2/32".to_string()), + address6: Some("fd00:66::2/128".to_string()), + gateway4: Some("10.66.0.1".to_string()), + gateway6: Some("fd00:66::1".to_string()), + server_public_key: hex::encode([0xab; 32]), + endpoint: "rs1.example:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + gateways: vec!["203.0.113.1".to_string()], + guests: vec![DesiredGuest { + address: "203.0.113.5/32".to_string(), + gateway: "203.0.113.1".to_string(), + mac: Some("aa:bb:cc:dd:ee:ff".to_string()), + }], + } +} + +fn key() -> NodeKey { + let dir = tempfile::tempdir().unwrap(); + wgkey::load_or_generate(dir.path()).unwrap() +} + +fn cidr(value: &str) -> IpNetwork { + value.parse().unwrap() +} + +/// A machine with nothing configured must end up with the whole data plane. +/// Any one part missing is a customer with no network. +#[tokio::test] +async fn a_bare_machine_gets_the_whole_data_plane() { + let kernel = FakeKernel::new(); + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + assert!(!changed.is_empty()); + + assert_eq!( + kernel.addresses_of(TUNNEL_INTERFACE), + vec![cidr("10.66.0.2/32"), cidr("fd00:66::2/128")] + ); + assert_eq!( + kernel.link_state(TUNNEL_INTERFACE).await.unwrap(), + (true, Some(1420)) + ); + // The bridge carries the same payload, so it takes the same MTU: a guest + // sending 1500 bytes into a 1420-byte tunnel opens a connection and then + // hangs on the first large transfer. + assert_eq!( + kernel.link_state(GUEST_BRIDGE).await.unwrap(), + (true, Some(1420)) + ); + + // Everything goes up the tunnel: the guests use LNVPS addresses, so no + // traffic of theirs belongs anywhere else. + let settings = kernel.wg_settings().unwrap(); + assert_eq!(settings.allowed_ips, vec![cidr("0.0.0.0/0"), cidr("::/0")]); + assert_eq!(settings.keepalive, Some(25)); + assert_eq!(settings.endpoint, "rs1.example:51820"); + + let mut tunnel_routes = kernel.routes_of(TUNNEL_INTERFACE); + tunnel_routes.sort(); + assert_eq!(tunnel_routes, vec![cidr("0.0.0.0/0"), cidr("::/0")]); + + // The gateway belongs to the range, not the node, and is held as a host + // address so the node answers for it without claiming the rest of the + // range is local. + assert_eq!( + kernel.addresses_of(GUEST_BRIDGE), + vec![cidr("203.0.113.1/32")] + ); + assert_eq!(kernel.routes_of(GUEST_BRIDGE), vec![cidr("203.0.113.5/32")]); + + // The guest thinks its neighbours are on-link and will ARP for them; proxy + // ARP is what lets the node answer and pull that traffic up the tunnel. + assert_eq!( + kernel.sysctl_value(&format!("net/ipv4/conf/{GUEST_BRIDGE}/proxy_arp")), + Some("1".to_string()) + ); + assert_eq!( + kernel.sysctl_value(&format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp")), + Some("1".to_string()) + ); + assert_eq!( + kernel.sysctl_value("net/ipv4/ip_forward"), + Some("1".to_string()) + ); + assert_eq!( + kernel.sysctl_value("net/ipv6/conf/all/forwarding"), + Some("1".to_string()) + ); +} + +/// A machine that is already right must be left alone. This runs every minute +/// on hardware LNVPS does not own; churn there is a tunnel that flaps. +#[tokio::test] +async fn a_correct_machine_is_not_touched_again() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + // Configuring WireGuard is stated unconditionally — the kernel API takes + // the whole interface and there is nothing to compare a private key + // against — but nothing else may move. + assert_eq!(changed, vec![format!("configured {TUNNEL_INTERFACE}")]); + assert_eq!(kernel.addresses_of(TUNNEL_INTERFACE).len(), 2); + assert_eq!(kernel.routes_of(GUEST_BRIDGE), vec![cidr("203.0.113.5/32")]); +} + +/// A peer that is not the route server has no business on this interface — +/// most likely a stale key from a re-key, still able to send traffic the node +/// would treat as LNVPS's. +#[tokio::test] +async fn a_stale_peer_is_removed() { + let kernel = FakeKernel::new(); + kernel.create_wireguard(TUNNEL_INTERFACE).await.unwrap(); + kernel + .wireguard + .lock() + .unwrap() + .get_mut(TUNNEL_INTERFACE) + .unwrap() + .peers + .push(("c3RyYXk=".to_string(), None)); + + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + assert!( + changed + .iter() + .any(|c| c.contains("removed stale peer c3RyYXk=")), + "{changed:?}" + ); + let peers = kernel + .wireguard_state(TUNNEL_INTERFACE) + .await + .unwrap() + .unwrap() + .peers; + assert_eq!(peers.len(), 1); +} + +/// A guest that has been deleted or moved must stop being routed here at once: +/// its address goes back in the pool and may already be somebody else's. +#[tokio::test] +async fn a_departed_guest_stops_being_routed() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + kernel + .add_route(cidr("203.0.113.9/32"), GUEST_BRIDGE) + .await + .unwrap(); + + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + assert!( + changed + .iter() + .any(|c| c.contains("unrouted 203.0.113.9/32")), + "{changed:?}" + ); + assert_eq!(kernel.routes_of(GUEST_BRIDGE), vec![cidr("203.0.113.5/32")]); + // The bridge's own gateway is an address, not a route, so a sweep of + // departed guests cannot take the bridge's addressing with it. + assert_eq!( + kernel.addresses_of(GUEST_BRIDGE), + vec![cidr("203.0.113.1/32")] + ); +} + +/// An address that is no longer ours goes, but the kernel's own link-local +/// address stays: removing it would break the interface, on every refresh. +#[tokio::test] +async fn a_stale_address_goes_and_the_kernels_own_stays() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + kernel + .add_address(TUNNEL_INTERFACE, cidr("10.66.0.9/32")) + .await + .unwrap(); + kernel + .add_address(TUNNEL_INTERFACE, cidr("fe80::1/64")) + .await + .unwrap(); + + apply(&kernel, &desired(), &key()).await.unwrap(); + let addresses = kernel.addresses_of(TUNNEL_INTERFACE); + assert!(!addresses.contains(&cidr("10.66.0.9/32")), "{addresses:?}"); + assert!(addresses.contains(&cidr("fe80::1/64")), "{addresses:?}"); +} + +/// A single-stack pool must not produce a default route for the family it has +/// no address in: that route black-holes traffic instead of leaving the +/// machine's own routing to handle it. +#[tokio::test] +async fn a_single_stack_tunnel_only_routes_its_own_family() { + let kernel = FakeKernel::new(); + let mut plane = desired(); + plane.tunnel.address6 = None; + apply(&kernel, &plane, &key()).await.unwrap(); + assert_eq!(kernel.routes_of(TUNNEL_INTERFACE), vec![cidr("0.0.0.0/0")]); +} + +/// A knob this kernel does not have is skipped, not fatal: IPv6 can be compiled +/// out, and a node with no IPv6 guests is still a working node. +#[tokio::test] +async fn a_kernel_without_ipv6_still_configures() { + let kernel = FakeKernel { + missing: vec![ + "net/ipv6/conf/all/forwarding".to_string(), + format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp"), + ], + ..FakeKernel::new() + }; + apply(&kernel, &desired(), &key()).await.unwrap(); + assert_eq!( + kernel.sysctl_value("net/ipv4/ip_forward"), + Some("1".to_string()) + ); +} + +/// An address LNVPS sent that is not an address is reported against the value: +/// the node has to say which part of the document it could not use. +#[tokio::test] +async fn a_malformed_address_is_reported() { + let kernel = FakeKernel::new(); + let mut plane = desired(); + plane.tunnel.address4 = Some("not-an-address".to_string()); + let err = apply(&kernel, &plane, &key()).await.unwrap_err(); + assert!(format!("{err:#}").contains("not-an-address"), "{err:#}"); + + let mut plane = desired(); + plane.gateways = vec!["also-not".to_string()]; + let err = apply(&kernel, &plane, &key()).await.unwrap_err(); + assert!(format!("{err:#}").contains("also-not"), "{err:#}"); +} + +/// Observation reads the machine rather than remembering what was applied: the +/// point of observing is to catch the case where the two disagree. +#[tokio::test] +async fn observation_reports_what_the_machine_has() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + + // `wg0` comes up happily with a peer that never answers, so an interface + // that has never handshaken is configured, not working. + let state = observe(&kernel).await.unwrap(); + assert!(state.tunnel_up); + assert_eq!(state.tunnel_mtu, Some(1420)); + assert_eq!(state.last_handshake_secs, None); + assert!(state.bridge_up); + assert!(state.forwarding4 && state.forwarding6); + assert_eq!(state.routed_guests, 1); + assert!( + !state.healthy(), + "a tunnel that never handshook is not healthy" + ); + + // Once the route server has answered, it is. + kernel + .wireguard + .lock() + .unwrap() + .get_mut(TUNNEL_INTERFACE) + .unwrap() + .peers = vec![("peer".to_string(), Some(12))]; + let state = observe(&kernel).await.unwrap(); + assert_eq!(state.last_handshake_secs, Some(12)); + assert!(state.healthy()); +} + +/// A machine with nothing configured reports nothing configured rather than +/// failing: "not set up yet" is a state the health gate has to be able to read. +#[tokio::test] +async fn an_unconfigured_machine_observes_cleanly() { + let kernel = FakeKernel::new(); + let state = observe(&kernel).await.unwrap(); + assert_eq!(state, DataPlaneState::default()); + assert!(!state.healthy()); +} + +/// `/proc/sys` is read and written as files rather than through the `sysctl` +/// binary: one less program a node must have installed, and a write that either +/// happens or says why. +#[test] +fn kernel_knobs_are_files() { + let dir = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(dir.path().join("net/ipv4")).unwrap(); + std::fs::write(dir.path().join("net/ipv4/ip_forward"), "0\n").unwrap(); + + assert_eq!( + read_sysctl(dir.path(), "net/ipv4/ip_forward").unwrap(), + Some("0\n".to_string()) + ); + write_sysctl(dir.path(), "net/ipv4/ip_forward", "1").unwrap(); + assert_eq!( + read_sysctl(dir.path(), "net/ipv4/ip_forward").unwrap(), + Some("1".to_string()) + ); + + // A knob that is not there is a fact about the machine, not a failure. + assert_eq!(read_sysctl(dir.path(), "net/ipv6/absent").unwrap(), None); + assert!(write_sysctl(dir.path(), "net/ipv6/absent", "1").is_err()); +} diff --git a/lnvps_node/src/netns.rs b/lnvps_node/src/netns.rs new file mode 100644 index 00000000..53748d1e --- /dev/null +++ b/lnvps_node/src/netns.rs @@ -0,0 +1,217 @@ +//! The network namespace the node's data plane lives in. +//! +//! A marketplace node is somebody else's machine, and often not only an LNVPS +//! node. Configuring the data plane in the machine's own namespace means taking +//! their default route, turning on forwarding machine-wide, and putting guest +//! routes and proxy-ARP settings in the tables their own tooling manages. All +//! of that is rude, and some of it is dangerous. +//! +//! So the data plane gets a namespace of its own: +//! +//! - **Their default route stays theirs.** Ours is inside, where only guest +//! traffic sees it. +//! - **Guests cannot reach the operator's network at all.** Not because a +//! firewall rule says so — a rule can be mis-ordered, flushed by their +//! tooling, or forgotten on one address family — but because the namespace +//! holds no interface that leads there. +//! - **A tunnel that is down means no path at all.** Without this, a stray +//! route sends customer traffic out the operator's uplink sourced from LNVPS +//! addresses, which looks like spoofing to their upstream and can get *their* +//! connection null-routed. +//! - **Kernel knobs are scoped.** `ip_forward`, `proxy_arp` and `proxy_ndp` are +//! per-namespace, so LNVPS stops editing settings on their machine. +//! +//! The one thing that must stay outside is the tunnel's own UDP socket: a +//! WireGuard interface keeps its socket in the namespace it was *created* in, +//! so `wg0` is created in the machine's namespace and then moved into this one. +//! The encrypted outer traffic still leaves through the operator's uplink, +//! while the inner interface — and everything routed over it — is isolated. + +use std::fs; +use std::os::fd::{AsFd, AsRawFd, OwnedFd}; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result}; +use nix::mount::{MntFlags, MsFlags, mount, umount2}; +use nix::sched::{CloneFlags, setns, unshare}; + +/// The namespace the data plane lives in. +pub const NAMESPACE: &str = "lnvps"; + +/// Where namespaces are pinned, matching iproute2 so `ip netns exec lnvps …` +/// works for an operator debugging their machine. +const NETNS_DIR: &str = "/run/netns"; + +/// The pinned path for a namespace. +pub fn path(root: &Path, name: &str) -> PathBuf { + root.join(name) +} + +/// Create the namespace if it does not exist, and return a handle to it. +/// +/// Pinned as a bind mount, exactly as `ip netns add` does it, so the namespace +/// outlives the daemon: a node restarting must not take its customers' network +/// with it, and an operator must be able to look inside with the tools they +/// already have. +pub fn ensure(root: &Path, name: &str) -> Result { + let pinned = path(root, name); + if pinned.exists() { + return Handle::open(&pinned); + } + + fs::create_dir_all(root) + .with_context(|| format!("Cannot create the namespace directory {}", root.display()))?; + fs::File::create(&pinned).with_context(|| { + format!( + "Cannot create the namespace mount point {}", + pinned.display() + ) + })?; + + // Done on a thread: unsharing changes the *calling thread's* namespace, and + // the daemon's other threads must keep the machine's own network — that is + // how it goes on reaching LNVPS while the tunnel is being built. + let pinned_for_thread = pinned.clone(); + std::thread::spawn(move || -> Result<()> { + unshare(CloneFlags::CLONE_NEWNET).context("Cannot create a network namespace")?; + // `/proc/thread-self`, not `/proc/self`: in a process with more than + // one thread, `/proc/self/ns/net` is the *process's* namespace, which + // is the one this thread just stopped being in. Pinning that would + // produce a "namespace" that is the machine's own network, and every + // isolation this module exists for would silently not happen. + mount( + Some(Path::new("/proc/thread-self/ns/net")), + &pinned_for_thread, + None::<&str>, + MsFlags::MS_BIND, + None::<&str>, + ) + .with_context(|| { + format!( + "Cannot pin the namespace at {}", + pinned_for_thread.display() + ) + })?; + Ok(()) + }) + .join() + .map_err(|_| anyhow::anyhow!("The thread creating the namespace panicked"))??; + + Handle::open(&pinned) +} + +/// Remove a pinned namespace. Used by tests and teardown, not in normal running. +pub fn remove(root: &Path, name: &str) -> Result<()> { + let pinned = path(root, name); + if !pinned.exists() { + return Ok(()); + } + // Unmounting drops the namespace once nothing else holds it; the file is + // then just a file. + let _ = umount2(&pinned, MntFlags::MNT_DETACH); + fs::remove_file(&pinned) + .with_context(|| format!("Cannot remove the namespace file {}", pinned.display())) +} + +/// The default pinned location, used outside tests. +pub fn ensure_default() -> Result { + ensure(Path::new(NETNS_DIR), NAMESPACE) +} + +/// An open network namespace. +#[derive(Debug)] +pub struct Handle { + fd: OwnedFd, + path: PathBuf, +} + +impl Handle { + /// Open an already-pinned namespace. + pub fn open(pinned: &Path) -> Result { + let file = fs::File::open(pinned) + .with_context(|| format!("Cannot open the namespace at {}", pinned.display()))?; + Ok(Self { + fd: OwnedFd::from(file), + path: pinned.to_path_buf(), + }) + } + + /// Where this namespace is pinned. + pub fn pinned_at(&self) -> &Path { + &self.path + } + + /// The descriptor, for the netlink call that moves an interface in here. + pub fn as_raw_fd(&self) -> std::os::fd::RawFd { + self.fd.as_fd().as_raw_fd() + } + + /// Run `f` with this namespace as the current one. + /// + /// On a thread of its own, because namespace membership is per-thread: the + /// daemon's other threads keep the machine's own network, which is what + /// lets it go on reaching LNVPS while the tunnel is down or being built. + /// The thread is scoped and joined here, so it ends inside the namespace + /// rather than being returned to a pool carrying it. + pub fn enter(&self, f: F) -> Result + where + F: FnOnce() -> Result + Send, + T: Send, + { + let target = self.fd.as_fd().as_raw_fd(); + std::thread::scope(|scope| { + scope + .spawn(move || -> Result { + setns_fd(target).context("Cannot enter the data plane namespace")?; + f() + }) + .join() + .map_err(|_| anyhow::anyhow!("A thread entering the namespace panicked"))? + }) + } +} + +/// `setns` on a raw fd, restricted to the network namespace. +fn setns_fd(fd: std::os::fd::RawFd) -> Result<()> { + // SAFETY: the fd is owned by the caller for the duration of the call. + let borrowed = unsafe { std::os::fd::BorrowedFd::borrow_raw(fd) }; + setns(borrowed, CloneFlags::CLONE_NEWNET)?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The whole point of the module: a thread inside the namespace must see a + /// different network from the machine's, and the namespace must outlive the + /// thread that made it so a daemon restart does not take the data plane + /// with it. + #[test] + #[ignore = "requires root; run with scripts/tunnel-e2e.sh"] + fn a_namespace_isolates_and_persists() { + let root = tempfile::tempdir().unwrap(); + let handle = ensure(root.path(), "lnvps-test").unwrap(); + + let outside = std::fs::read_link("/proc/thread-self/ns/net").unwrap(); + let inside = handle + .enter(|| Ok(std::fs::read_link("/proc/thread-self/ns/net")?)) + .unwrap(); + assert_ne!( + outside, inside, + "the thread stayed in the machine's network" + ); + + // The daemon's own threads keep the machine's network, which is what + // lets it go on reaching LNVPS while the tunnel is down. + assert_eq!( + std::fs::read_link("/proc/thread-self/ns/net").unwrap(), + outside + ); + + // Pinned, so it survives this process. + assert!(path(root.path(), "lnvps-test").exists()); + remove(root.path(), "lnvps-test").unwrap(); + assert!(!path(root.path(), "lnvps-test").exists()); + } +} diff --git a/lnvps_node/tests/control_https.rs b/lnvps_node/tests/control_https.rs index d15584e5..289e5422 100644 --- a/lnvps_node/tests/control_https.rs +++ b/lnvps_node/tests/control_https.rs @@ -32,7 +32,13 @@ async fn start_node(keys: &Keys, state_dir: &std::path::Path) -> (SocketAddr, Ve let node_tls = tls::load_or_generate(state_dir, Some(addr.ip())).unwrap(); let (cert, fingerprint) = (node_tls.cert_pem.clone(), node_tls.fingerprint.clone()); - let state = Arc::new(ControlState::new(keys.public_key(), addr)); + // The data plane this reports on is the machine's; these tests are about + // TLS and authentication, so an unconfigured machine is the honest answer. + let state = Arc::new(ControlState::new( + keys.public_key(), + addr, + Arc::new(lnvps_node::net::UnavailableKernel), + )); tokio::spawn(async move { serve(state, addr, node_tls).await }); // Wait for the listener rather than sleeping a fixed time. diff --git a/scripts/tunnel-e2e.sh b/scripts/tunnel-e2e.sh new file mode 100755 index 00000000..fc7ff6d6 --- /dev/null +++ b/scripts/tunnel-e2e.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# tunnel-e2e.sh — Run the marketplace tunnel harness: both ends, real kernel. +# +# The harness (lnvps_e2e/tests/tunnel_netns.rs) builds a route server and a node +# out of network namespaces, configures each end with the real production code +# paths, and pings across the tunnel — including to a guest sitting behind the +# node, which is the path a customer's traffic takes. +# +# It needs root (namespaces, veth, WireGuard), so the tests are #[ignore]d and +# only run from here. +# +# Usage: +# ./scripts/tunnel-e2e.sh [--filter NAME] + +set -euo pipefail + +FILTER="" +while [[ $# -gt 0 ]]; do + case "$1" in + --filter) FILTER="$2"; shift 2 ;; + --) shift; break ;; + *) echo "Unknown option: $1" >&2; exit 1 ;; + esac +done + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$(cd "$SCRIPT_DIR/.." && pwd)" + +# Built as the invoking user so the build cache and toolchain resolve against +# the normal $HOME; only the run needs to be privileged. +echo "=== Building the tunnel harness ===" +cargo test -p lnvps_e2e --test tunnel_netns --no-run + +CMD=(cargo test -p lnvps_e2e --test tunnel_netns -- --ignored --test-threads=1) +[[ -n "$FILTER" ]] && CMD+=("$FILTER") +[[ $# -gt 0 ]] && CMD+=("$@") + +echo "=== Running the tunnel harness as root ===" +if [[ "$(id -u)" -eq 0 ]]; then + "${CMD[@]}" +else + sudo -E "${CMD[@]}" +fi diff --git a/work/marketplace.md b/work/marketplace.md index 80aad514..2a694148 100644 --- a/work/marketplace.md +++ b/work/marketplace.md @@ -884,9 +884,38 @@ What the build settled beyond the plan: - **`lnvps-node dataplane observe` deliberately needs no credential**, because "what does this machine actually have?" is the question asked when something is already broken. -Testing note: `net.rs` runs commands through a `CommandRunner`, faked in tests to answer `ip` -and `wg` queries from a script and record everything it was asked to change. These commands run -as root on somebody else's hardware, so the exact command issued is the thing worth asserting. +Reworked during review, before merge: +- **Netlink, not `ip`.** The daemon speaks to the kernel directly (`rtnetlink` for links, + addresses and routes; the WireGuard netlink interface for the tunnel; `/proc/sys` for the + forwarding knobs). `ip` is a program that formats netlink messages and formats the answers + back into text for us to parse; going direct removes a dependency on iproute2's presence and + version, the output parsing that changes between releases, and gives kernel error codes + instead of a line of English on stderr. +- **The data plane lives in its own network namespace.** The first cut configured the + *machine's* network: it took the operator's default route and turned on forwarding + machine-wide, on hardware that is often not only an LNVPS node. Now `wg0` and `br-lnvps` live + in an `lnvps` namespace, so their default route stays theirs, the forwarding and proxy-ARP + knobs are ours alone, and guests cannot reach the operator's network — not because a rule + forbids it, but because no interface leads there. A tunnel that is down means no path at all, + rather than customer traffic leaking out the operator's uplink sourced from LNVPS addresses, + which looks like spoofing to their upstream. `wg0` is created in the machine's namespace and + *moved*, because a WireGuard interface keeps its UDP socket where it was created — that is + what lets the encrypted outer traffic still use the operator's uplink. +- **The bridge name is no longer sent.** Both sides hold it as a constant, because the daemon + needs the name before it has ever spoken to LNVPS (`dataplane observe` takes no credential), + and a document that could name a different one would leave the node holding two answers. The + harness asserts the two constants agree. + +Testing note: the orchestration is tested against a fake kernel behind a `NetOps` trait — what +is worth asserting is what the node *decides*. Whether those decisions work is proven by +`lnvps_e2e/tests/tunnel_netns.rs`, which builds both ends in network namespaces and pings +across the tunnel, including to a guest behind the node. It found four bugs nothing else did: +a namespace pinned from `/proc/self/ns/net` (the *process's* namespace in a threaded program, +so every "isolated" interface silently landed in the operator's network), WireGuard netlink +calls made outside the namespace the interface had been moved into, local-table routes being +mistaken for strays and deleted — and, in already-merged 4b code, **the route server never +routing the pool's own block**, so a route server holding `10.66.0.1/16` answered "network is +unreachable" for every node in the pool. #### 4c1 — original scope - `GET /api/v1/node/dataplane` (node token): the desired state in one document — the tunnel From 00dd68f47796066393fd5023aaf9944e97022d0b Mon Sep 17 00:00:00 2001 From: v0l Date: Fri, 7 Aug 2026 10:18:37 +0100 Subject: [PATCH 3/3] fix(marketplace): name the node's tunnel wgln0, not wg0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The interface is created in the *machine's* network namespace before being moved into the data plane's, so its name has to be one the operator is not already using. `wg0` is the default name for every WireGuard tutorial, VPN and mesh on Linux: on a node whose operator already has one, creating ours either fails outright or — far worse — adopts theirs and moves it into our namespace, taking their VPN down with no indication why. `wgln0` matches the prefix LNVPS already uses for route-server interfaces (`wgln`), so a managed interface is recognisable as ours wherever it turns up, on either end of the tunnel. The end-to-end harness now puts an operator's own `wg0` on the machine before the node configures itself, and checks it is still there afterwards. --- docs/agents/e2e-tests.md | 2 +- lnvps_e2e/tests/tunnel_netns.rs | 33 +++++++++++++++++++++++---------- lnvps_node/config.example.yaml | 5 ++++- lnvps_node/src/config.rs | 4 ++-- lnvps_node/src/net.rs | 12 +++++++++--- lnvps_node/src/net/tests.rs | 2 +- lnvps_node/src/netns.rs | 3 ++- work/marketplace.md | 14 ++++++++++---- 8 files changed, 52 insertions(+), 23 deletions(-) diff --git a/docs/agents/e2e-tests.md b/docs/agents/e2e-tests.md index 7b901314..f0e6ece1 100644 --- a/docs/agents/e2e-tests.md +++ b/docs/agents/e2e-tests.md @@ -325,7 +325,7 @@ packets across it. ```text [rs netns] [machine netns] [lnvps netns] [guest netns] - wgln <══ WireGuard ══> wg0 created here, then ══> wg0 veth + wgln <══ WireGuard ══> wgln0 created here, then ═> wgln0 veth 10.66.0.1/24 its UDP socket stays here 10.66.0.2/32 br-lnvps ── 203.0.113.5/24 ``` diff --git a/lnvps_e2e/tests/tunnel_netns.rs b/lnvps_e2e/tests/tunnel_netns.rs index bff630b7..e3a91c72 100644 --- a/lnvps_e2e/tests/tunnel_netns.rs +++ b/lnvps_e2e/tests/tunnel_netns.rs @@ -9,7 +9,7 @@ //! //! ```text //! [rs netns] [test machine's netns] [lnvps netns] [guest netns] -//! wgln <══ WireGuard ══> wg0 created here, then ══> wg0 veth +//! wgln <══ WireGuard ══> wgln0 created here, then ═> wgln0 veth //! 10.66.0.1/24 its UDP socket stays here 10.66.0.2/32 br-lnvps ── 203.0.113.5/24 //! │ 203.0.113.1/32 //! rs_up veth ────────────────── node_up veth @@ -17,7 +17,7 @@ //! ``` //! //! The shape is the production one, including the part that is easy to get -//! wrong: `wg0` is created in the machine's own namespace so its UDP socket can +//! wrong: `wgln0` is created in the machine's own namespace so its UDP socket can //! reach the route server through the operator's uplink, and is then moved into //! the LNVPS namespace so that everything carried *over* the tunnel is isolated //! from the operator's network. @@ -116,10 +116,11 @@ impl Topology { let _ = Command::new("ip") .args(["netns", "delete", &self.dataplane]) .output(); - let _ = Command::new("ip") - .args(["link", "del", "e2e-node"]) - .output(); - let _ = Command::new("ip").args(["link", "del", "wg0"]).output(); + // Anything the node's code created in the machine's namespace, which + // is where a half-finished run leaves it. + for link in ["e2e-node", "e2e-guest", "wgln0", "wg0", "br-lnvps"] { + let _ = Command::new("ip").args(["link", "del", link]).output(); + } } } @@ -325,7 +326,7 @@ async fn a_guest_behind_a_node_is_reachable_from_the_route_server() -> Result<() .in_rs(&["ping", "-c", "3", "-W", "5", GUEST_ADDRESS]) .context("the route server could not reach a guest behind the node")?; - // The node reports itself healthy only once that has happened: `wg0` comes + // The node reports itself healthy only once that has happened: WireGuard comes // up perfectly happily with a peer that never answers. let state = lnvps_node::net::observe(&kernel).await?; assert!(state.tunnel_up, "{state:?}"); @@ -353,6 +354,12 @@ async fn the_operators_machine_keeps_its_own_network() -> Result<()> { let default_before = run("ip", &["-j", "route", "show", "default"])?; + // The operator's own WireGuard interface, which is why the node's is not + // called `wg0`: the node creates its interface in *this* namespace before + // moving it, so a collision here would either fail outright or, worse, + // adopt somebody's VPN and move it out from under them. + run("ip", &["link", "add", "wg0", "type", "wireguard"])?; + let kernel = lnvps_node::net::Kernel::in_namespace(topology.open_dataplane()?)?; lnvps_node::net::apply( &kernel, @@ -374,17 +381,23 @@ async fn the_operators_machine_keeps_its_own_network() -> Result<()> { ) .await?; + // The operator's interface is still theirs, still where they left it. + assert!( + run("ip", &["link", "show", "wg0"]).is_ok(), + "the operator's own wg0 was taken or destroyed" + ); + // The interfaces exist in the namespace and nowhere else. Asserted first, // because if this is wrong every later assertion is wrong for the same // reason and this is the one that says why. assert!( - run("ip", &["link", "show", "wg0"]).is_err(), - "wg0 is still in the machine's namespace: {}", + run("ip", &["link", "show", "wgln0"]).is_err(), + "the node's interface is still in the machine's namespace: {}", run("ip", &["link", "show"]).unwrap_or_default() ); assert!( topology - .in_dataplane(&["ip", "link", "show", "wg0"]) + .in_dataplane(&["ip", "link", "show", "wgln0"]) .is_ok() ); assert!(run("ip", &["link", "show", "br-lnvps"]).is_err()); diff --git a/lnvps_node/config.example.yaml b/lnvps_node/config.example.yaml index 07182eb0..29116241 100644 --- a/lnvps_node/config.example.yaml +++ b/lnvps_node/config.example.yaml @@ -37,4 +37,7 @@ control: # real addresses at startup. listen: "10.66.0.2" port: 8890 - tunnel-interface: "wg0" + # Named with the LNVPS prefix rather than wg0: the daemon creates it on the + # machine before moving it into its own network namespace, and an operator's + # own wg0 is a name it must not collide with. + tunnel-interface: "wgln0" diff --git a/lnvps_node/src/config.rs b/lnvps_node/src/config.rs index 1264fafd..12b0688f 100644 --- a/lnvps_node/src/config.rs +++ b/lnvps_node/src/config.rs @@ -80,7 +80,7 @@ fn default_control_port() -> u16 { } fn default_tunnel_interface() -> String { - "wg0".to_string() + crate::net::TUNNEL_INTERFACE.to_string() } impl NodeConfig { @@ -257,7 +257,7 @@ control: let control = NodeConfig::load(&path).unwrap().control.unwrap(); assert_eq!(control.listen, v4("10.66.0.1")); assert_eq!(control.port, 8890); - assert_eq!(control.tunnel_interface, "wg0"); + assert_eq!(control.tunnel_interface, "wgln0"); } /// A typo in a key must not be silently ignored: a misspelled `listen` diff --git a/lnvps_node/src/net.rs b/lnvps_node/src/net.rs index d70be8e3..7ebdcaad 100644 --- a/lnvps_node/src/net.rs +++ b/lnvps_node/src/net.rs @@ -35,7 +35,13 @@ use serde::{Deserialize, Serialize}; use crate::wgkey::{self, NodeKey}; /// The tunnel interface the node terminates its data plane on. -pub const TUNNEL_INTERFACE: &str = "wg0"; +/// +/// `wgln0`, not `wg0`: the interface is created in the *machine's* namespace +/// before being moved into the data plane's, and an operator's own `wg0` — a +/// VPN, a mesh, anything — is a name we would collide with there. The `wgln` +/// prefix is the same one LNVPS uses for its route-server interfaces, so a +/// managed interface is recognisable as ours wherever it turns up. +pub const TUNNEL_INTERFACE: &str = "wgln0"; /// The bridge guests sit on. /// @@ -164,7 +170,7 @@ pub struct DataPlaneState { impl DataPlaneState { /// Whether this node can carry a customer. /// - /// A handshake is required, not just an interface: `wg0` comes up happily + /// A handshake is required, not just an interface: WireGuard comes up happily /// with a peer that never answers, and a node in that state looks /// configured while being unreachable. pub fn healthy(&self) -> bool { @@ -190,7 +196,7 @@ pub async fn apply( Ok(changed) } -/// Bring up `wg0` and point the default route down it. +/// Bring up the tunnel interface and point the default route down it. async fn apply_tunnel( ops: &dyn NetOps, desired: &DesiredDataPlane, diff --git a/lnvps_node/src/net/tests.rs b/lnvps_node/src/net/tests.rs index 5b928314..dc68f194 100644 --- a/lnvps_node/src/net/tests.rs +++ b/lnvps_node/src/net/tests.rs @@ -447,7 +447,7 @@ async fn observation_reports_what_the_machine_has() { let kernel = FakeKernel::new(); apply(&kernel, &desired(), &key()).await.unwrap(); - // `wg0` comes up happily with a peer that never answers, so an interface + // WireGuard comes up happily with a peer that never answers, so an interface // that has never handshaken is configured, not working. let state = observe(&kernel).await.unwrap(); assert!(state.tunnel_up); diff --git a/lnvps_node/src/netns.rs b/lnvps_node/src/netns.rs index 53748d1e..6fc33e82 100644 --- a/lnvps_node/src/netns.rs +++ b/lnvps_node/src/netns.rs @@ -23,7 +23,8 @@ //! //! The one thing that must stay outside is the tunnel's own UDP socket: a //! WireGuard interface keeps its socket in the namespace it was *created* in, -//! so `wg0` is created in the machine's namespace and then moved into this one. +//! so the tunnel interface is created in the machine's namespace and then moved +//! into this one. //! The encrypted outer traffic still leaves through the operator's uplink, //! while the inner interface — and everything routed over it — is isolated. diff --git a/work/marketplace.md b/work/marketplace.md index 2a694148..98464e6f 100644 --- a/work/marketplace.md +++ b/work/marketplace.md @@ -99,7 +99,7 @@ Node is a **client**, never a server: ### Data plane (all traffic over WireGuard) ``` -guest VM ─ tap ─ br-lnvps (no operator uplink) ─ wg0 ─┐ +guest VM ─ tap ─ br-lnvps (no operator uplink) ─ wgln0 ─┐ │ WG (UDP) operator NAT / any ISP │ ▼ @@ -744,7 +744,7 @@ What the build settled beyond the plan: constraint, which is exactly the pool-less case. The mock mirrors it. - **A node takes one address, not a link** (revised during 4b; 4a shipped /31s and /127s). WireGuard is layer 3 and point-to-point, with no ARP and no on-link requirement, so the node - needs no gateway of its own — `ip route add default dev wg0` is enough. A /31 therefore spent + needs no gateway of its own — `ip route add default dev wgln0` is enough. A /31 therefore spent two addresses describing something that needs one, and worse, forced the route server to hold one address per node on a single interface: a /16 pool with a thousand nodes meant a thousand addresses on `wgln`, re-parsed out of `ip addr show` on every reconcile. The route server @@ -868,7 +868,7 @@ What the build settled beyond the plan: - **The bridge takes the tunnel's MTU.** A guest sending 1500 bytes into a 1420-byte tunnel gets a connection that opens and then hangs on the first large transfer — the worst failure shape there is, because everything looks fine until it does not. -- **A peer that is not the route server is removed from `wg0`**, most likely a stale key left by +- **A peer that is not the route server is removed from the tunnel interface**, most likely a stale key left by a re-key, which would otherwise still be able to send traffic the node treats as LNVPS's. - **Routes for departed guests are swept**, since a released address goes straight back in the pool and may already be somebody else's; the bridge's own gateway addresses are excluded from @@ -876,7 +876,7 @@ What the build settled beyond the plan: - **Observation reads the machine, never a cache.** `/api/v1/status` runs the queries on demand: a cached answer reports that the tunnel was up once, which is exactly what the health gate must not accept. A tunnel that has never handshaken is reported as configured but not - working, because `wg0` comes up perfectly happily with a peer that never answers. + working, because WireGuard comes up perfectly happily with a peer that never answers. - **The node generates its key in-process** rather than shelling out to `wg genkey`, so a missing `wg` fails when the interface is configured, with that error. The private key reaches `wg` as a **path**, never an argument: arguments are visible in `ps` to every user on the @@ -901,6 +901,12 @@ Reworked during review, before merge: which looks like spoofing to their upstream. `wg0` is created in the machine's namespace and *moved*, because a WireGuard interface keeps its UDP socket where it was created — that is what lets the encrypted outer traffic still use the operator's uplink. +- **The node's tunnel is `wgln0`, not `wg0`.** It is created in the *machine's* namespace + before being moved into the data plane's, so the name has to be one an operator is not + already using — a VPN, a mesh, anything called `wg0` would either fail the creation or, worse, + be adopted and moved out from under them. The `wgln` prefix is the same one the route server + uses, so a managed interface is recognisable as LNVPS's wherever it appears. The harness + proves it by putting an operator's own `wg0` on the machine first and checking it survives. - **The bridge name is no longer sent.** Both sides hold it as a constant, because the daemon needs the name before it has ever spoken to LNVPS (`dataplane observe` takes no credential), and a document that could name a different one would leave the node holding two answers. The