diff --git a/API_CHANGELOG.md b/API_CHANGELOG.md index 23886f5e..6bae5292 100644 --- a/API_CHANGELOG.md +++ b/API_CHANGELOG.md @@ -56,6 +56,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ### Added +- **The route server now routes its tunnel pool's own blocks.** An address on a point-to-point interface does not route the rest of its prefix, so a route server holding `10.66.0.1/16` answered "network is unreachable" for every node in that pool — the peers were configured correctly and unreachable. Found by an end-to-end harness that builds both ends of a tunnel and sends real packets, not by review. + +- **A node can fetch its whole data plane in one document** — `GET /api/v1/node/dataplane` (node token) returns the tunnel it already gets from `/node/tunnel`, plus the gateway addresses the node must answer for, and the guests assigned to it (address, gateway, MAC). One call rather than three because the node applies these together or not at all: a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list — an address not in it is not that node's to send from. The gateway is the one the guest was actually configured with, taken from its IP range: it belongs to the range rather than to the node, and the guest believes it is on-link, so the node has to answer for it rather than invent one. + - **A node takes one address, not a point-to-point link** — `address4`/`address6` in the tunnel response are now a `/32` and a `/128`, and `gateway4`/`gateway6` are one address shared by every node on the pool rather than a per-node link address. WireGuard is layer 3 and point-to-point: the node needs no gateway on its own side (`ip route add default dev wg0` suffices), so a `/31` spent two addresses describing something that needs one — and forced the route server to carry one address per node on a single interface, thousands of them on a /16 pool. Pool capacity is reported accordingly: a /24 places 253 nodes (256 less the block's network address, the route server's address after it, and the broadcast address) where it previously reported 128 links. - **A marketplace node's tunnel is now actually configured on the route server** — allocating a tunnel (`POST /api/v1/node/tunnel`) previously wrote down addresses and a key and stopped there; the node's peer is now pushed to the route server, so the tunnel carries traffic instead of only existing on paper. The peer's `AllowedIPs` is the node's own inner addresses plus exactly the guest addresses LNVPS assigned to it, which is also the anti-spoof boundary: WireGuard drops an inbound packet whose source is not on that list, so a node cannot source traffic as another node's customer. The route server takes an address on each point-to-point link, and each guest address is routed down the interface — `AllowedIPs` decides which *peer* a packet already headed for the tunnel belongs to, it does not put the packet there. Peers are also reconciled on the routine router poll, so an address assigned to a guest since the last push, a peer wiped by a route-server reboot, and a stale key nobody removed are all corrected without an admin doing anything; the reconcile reports what had drifted rather than quietly fixing it. The backing host stays **disabled** — a configured peer is still not a proven path, and the health gate that enables it comes with the node-side data plane. diff --git a/Cargo.lock b/Cargo.lock index 36ac2478..5efc91dd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -195,7 +195,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7af50940b73bf4e16c15c448a2b121c63f2d68e3e54b6a8731673cb4aa0cdff5" dependencies = [ "base64ct", - "blake2", + "blake2 0.11.0-rc.6", "cpufeatures 0.3.0", "password-hash 0.6.1", ] @@ -737,6 +737,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "blake2" +version = "0.10.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" +dependencies = [ + "digest 0.10.7", +] + [[package]] name = "blake2" version = "0.11.0-rc.6" @@ -1308,6 +1317,20 @@ dependencies = [ "subtle", ] +[[package]] +name = "curve25519-dalek" +version = "4.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" +dependencies = [ + "cfg-if", + "cpufeatures 0.2.17", + "curve25519-dalek-derive", + "fiat-crypto 0.2.9", + "rustc_version", + "subtle", +] + [[package]] name = "curve25519-dalek" version = "5.0.0" @@ -1318,7 +1341,7 @@ dependencies = [ "cpufeatures 0.3.0", "curve25519-dalek-derive", "digest 0.11.3", - "fiat-crypto", + "fiat-crypto 0.3.0", "rand_core 0.10.1", "rustc_version", "subtle", @@ -1409,6 +1432,55 @@ version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "092966b41edc516079bdf31ec78a2e0588d1d0c08f78b91d8307215928642b2b" +[[package]] +name = "defguard_boringtun" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6d920cd0791b2199308a3c8cc0d41b88d96e4c07d70e98b658a4ee72ad32ca7e" +dependencies = [ + "aead 0.5.2", + "base64 0.22.1", + "blake2 0.10.6", + "chacha20poly1305", + "hex", + "hmac 0.12.1", + "ip_network", + "ip_network_table", + "libc", + "nix 0.31.2", + "parking_lot", + "ring", + "socket2 0.6.4", + "thiserror 2.0.18", + "tracing", + "x25519-dalek 2.0.1", +] + +[[package]] +name = "defguard_wireguard_rs" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a09896853b7e5f2302c3e6c6786b3dc0433bc092cc9e3e647812d617d0c2a0ce" +dependencies = [ + "base64 0.22.1", + "defguard_boringtun", + "ipnet", + "libc", + "log 0.4.32", + "netlink-packet-core", + "netlink-packet-generic", + "netlink-packet-route 0.31.0", + "netlink-packet-utils", + "netlink-packet-wireguard", + "netlink-sys", + "regex", + "serde", + "thiserror 2.0.18", + "windows", + "wireguard-nt", + "x25519-dalek 3.0.0", +] + [[package]] name = "delegate" version = "0.13.5" @@ -1630,7 +1702,7 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ebaa1a2bf1290ab3bfe5a7b771d050ebffab2711c19a81691c683a5144a25de" dependencies = [ - "curve25519-dalek", + "curve25519-dalek 5.0.0", "ed25519", "rand_core 0.10.1", "serde", @@ -1907,6 +1979,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "fiat-crypto" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "28dea519a9695b9977216879a3ebfddf92f1c08c05d984f8996aecd6ecdc811d" + [[package]] name = "fiat-crypto" version = "0.3.0" @@ -2021,9 +2099,9 @@ dependencies = [ [[package]] name = "futures-channel" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" dependencies = [ "futures-core", "futures-sink", @@ -2031,9 +2109,9 @@ dependencies = [ [[package]] name = "futures-core" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" [[package]] name = "futures-executor" @@ -2059,15 +2137,15 @@ dependencies = [ [[package]] name = "futures-io" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" [[package]] name = "futures-macro" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" dependencies = [ "proc-macro2", "quote", @@ -2076,15 +2154,15 @@ dependencies = [ [[package]] name = "futures-sink" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" [[package]] name = "futures-task" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" [[package]] name = "futures-timer" @@ -2094,9 +2172,9 @@ checksum = "af43fadb8a98512d547e37b4e92e0ced13e205c061b87b4623eff01d918d6968" [[package]] name = "futures-util" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" dependencies = [ "futures-channel", "futures-core", @@ -2889,6 +2967,28 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "ip_network" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aa2f047c0a98b2f299aa5d6d7088443570faae494e9ae1305e48be000c9e0eb1" + +[[package]] +name = "ip_network_table" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4099b7cfc5c5e2fe8c5edf3f6f7adf7a714c9cc697534f63a5a5da30397cb2c0" +dependencies = [ + "ip_network", + "ip_network_table-deps-treebitmap", +] + +[[package]] +name = "ip_network_table-deps-treebitmap" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e537132deb99c0eb4b752f0346b6a836200eaaa3516dd7e5514b63930a09e5d" + [[package]] name = "ipconfig" version = "0.3.4" @@ -2896,7 +2996,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d40460c0ce33d6ce4b0630ad68ff63d6661961c48b6dba35e5a4d81cfb48222" dependencies = [ "socket2 0.6.4", - "widestring", + "widestring 1.2.1", "windows-registry", "windows-result", "windows-sys 0.61.2", @@ -3275,6 +3375,16 @@ version = "0.2.183" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + [[package]] name = "libm" version = "0.2.16" @@ -3507,7 +3617,7 @@ dependencies = [ "uuid", "virt", "wiremock", - "x25519-dalek", + "x25519-dalek 3.0.0", ] [[package]] @@ -3555,6 +3665,10 @@ dependencies = [ "chrono", "futures-util", "hex", + "lnvps_api", + "lnvps_api_common", + "lnvps_node", + "nix 0.30.1", "nostr 0.44.3", "p256 0.13.2", "rand_core 0.6.4", @@ -3564,6 +3678,7 @@ dependencies = [ "serde_json", "sha2 0.10.9", "sqlx", + "tempfile", "tokio", "tokio-tungstenite 0.28.0", "webauthn-rs", @@ -3605,18 +3720,28 @@ name = "lnvps_node" version = "0.4.7" dependencies = [ "anyhow", + "async-trait", "axum", "axum-server", "base64 0.22.1", "clap", "config", + "defguard_wireguard_rs", "env_logger", + "futures-util", + "hex", + "http", "if-addrs", + "ipnetwork", "lnvps_host_util", "log 0.4.32", + "netlink-packet-route 0.30.0", + "nix 0.30.1", "nostr 0.44.3", + "rand 0.9.4", "rcgen", "reqwest 0.12.28", + "rtnetlink", "rustls", "serde", "serde_json", @@ -3624,6 +3749,7 @@ dependencies = [ "tokio", "tokio-rustls", "tower", + "x25519-dalek 3.0.0", ] [[package]] @@ -3875,6 +4001,112 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0efe882e02d206d8d279c20eb40e03baf7cb5136a1476dc084a324fbc3ec42d" +[[package]] +name = "netlink-packet-core" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b897d7bd4f0af82e68d40d0344cf37e97f9c97ddf74a098de3e4da05e96ca395" +dependencies = [ + "paste", +] + +[[package]] +name = "netlink-packet-generic" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2f891b2e0054cac5a684a06628f59568f841c93da4e551239da6e518f539e775" +dependencies = [ + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-route" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "be8919612f6028ab4eacbbfe1234a9a43e3722c6e0915e7ff519066991905092" +dependencies = [ + "bitflags", + "libc", + "log 0.4.32", + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-route" +version = "0.31.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2288fcb784eb3defd5fb16f4c4160d5f477de192eac730f43e1d11c24d9a007" +dependencies = [ + "bitflags", + "libc", + "log 0.4.32", + "netlink-packet-core", +] + +[[package]] +name = "netlink-packet-utils" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3176f18d11a1ae46053e59ec89d46ba318ae1343615bd3f8c908bfc84edae35c" +dependencies = [ + "byteorder", + "pastey", + "thiserror 2.0.18", +] + +[[package]] +name = "netlink-packet-wireguard" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8901a50aa367f28d70e8c2a6b379f51efef1e5acebbe05f940ae00363d7d83fd" +dependencies = [ + "bitflags", + "libc", + "log 0.4.32", + "netlink-packet-core", + "netlink-packet-generic", +] + +[[package]] +name = "netlink-proto" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6f7398dddf5f152d2a91a2921a134c6097056e292c0d4b9906007855e7cece6" +dependencies = [ + "bytes", + "futures-channel", + "futures-util", + "log 0.4.32", + "netlink-packet-core", + "netlink-sys", + "thiserror 2.0.18", +] + +[[package]] +name = "netlink-sys" +version = "0.8.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd6c30ed10fa69cc491d491b85cc971f6bdeb8e7367b7cde2ee6cc878d583fae" +dependencies = [ + "bytes", + "futures-util", + "libc", + "log 0.4.32", + "tokio", +] + +[[package]] +name = "nix" +version = "0.30.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] + [[package]] name = "nix" version = "0.31.2" @@ -4258,7 +4490,7 @@ checksum = "d17b851e6b3e378ab4ecb07fa2ed23f4d15f075735f8fec9fa1e7bdce5f8301f" dependencies = [ "ecdsa 0.17.0", "elliptic-curve 0.14.1", - "fiat-crypto", + "fiat-crypto 0.3.0", "primefield", "primeorder 0.14.0", "sha2 0.11.0", @@ -4361,6 +4593,12 @@ dependencies = [ "phc", ] +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + [[package]] name = "pastey" version = "0.1.1" @@ -5331,6 +5569,24 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rtnetlink" +version = "0.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc19f84f710fa2f337617f9bc0400260a94224bde7bae28fd8879f3771ca5784" +dependencies = [ + "futures-channel", + "futures-util", + "log 0.4.32", + "netlink-packet-core", + "netlink-packet-route 0.30.0", + "netlink-proto", + "netlink-sys", + "nix 0.30.1", + "thiserror 1.0.69", + "tokio", +] + [[package]] name = "russh" version = "0.62.4" @@ -5347,7 +5603,7 @@ dependencies = [ "cipher 0.5.2", "crypto-bigint 0.7.5", "ctr 0.10.1", - "curve25519-dalek", + "curve25519-dalek 5.0.0", "data-encoding", "delegate", "der 0.8.1", @@ -5410,7 +5666,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3aec6cb630dbe85d72ffd7bcd95f07e1bd69f9f270ee8adfa1afe443a6331438" dependencies = [ "log 0.4.32", - "nix", + "nix 0.31.2", "ssh-encoding 0.3.0", "windows-sys 0.61.2", ] @@ -7474,6 +7730,12 @@ dependencies = [ "wasite", ] +[[package]] +name = "widestring" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c168940144dd21fd8046987c16a46a33d5fc84eec29ef9dcddc2ac9e31526b7c" + [[package]] name = "widestring" version = "1.2.1" @@ -7850,6 +8112,22 @@ dependencies = [ "memchr", ] +[[package]] +name = "wireguard-nt" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22b4dbcc6c93786cf22e420ef96e8976bfb92a455070282302b74de5848191f4" +dependencies = [ + "bitflags", + "getrandom 0.2.17", + "ipnet", + "libloading", + "log 0.4.32", + "thiserror 1.0.69", + "widestring 0.4.3", + "windows-sys 0.59.0", +] + [[package]] name = "wiremock" version = "0.6.5" @@ -7896,13 +8174,24 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" +[[package]] +name = "x25519-dalek" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7e468321c81fb07fa7f4c636c3972b9100f0346e5b6a9f2bd0603a52f7ed277" +dependencies = [ + "curve25519-dalek 4.1.3", + "rand_core 0.6.4", +] + [[package]] name = "x25519-dalek" version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e7e8131a03190127fb2263afc72b322ecadae46b6ff8c6f399ff5d02f5559af6" dependencies = [ - "curve25519-dalek", + "curve25519-dalek 5.0.0", + "getrandom 0.4.3", "rand_core 0.10.1", "zeroize", ] diff --git a/docs/agents/e2e-tests.md b/docs/agents/e2e-tests.md index de9d56bd..f0e6ece1 100644 --- a/docs/agents/e2e-tests.md +++ b/docs/agents/e2e-tests.md @@ -315,3 +315,51 @@ The `.github/workflows/e2e.yml` workflow runs E2E tests on every pull request. I | `.github/e2e/api-config.yaml` | User API config template (DB URL replaced at runtime) | | `.github/e2e/admin-config.yaml` | Admin API config template (DB URL replaced at runtime) | | `.github/e2e/wait-for-lnd.sh` | Script to wait for LND readiness and mine initial blocks | + +## Marketplace tunnel harness (`tests/tunnel_netns.rs`) + +A second kind of end-to-end test lives in the same crate and shares none of the +above infrastructure: no API server, no database, no docker. It builds **both +ends of a marketplace tunnel** out of Linux network namespaces and sends real +packets across it. + +```text + [rs netns] [machine netns] [lnvps netns] [guest netns] + wgln <══ WireGuard ══> wgln0 created here, then ═> wgln0 veth + 10.66.0.1/24 its UDP socket stays here 10.66.0.2/32 br-lnvps ── 203.0.113.5/24 +``` + +Both ends run production code: the route server is configured through +`LinuxSshRouter` with its command transport pointed at `ip netns exec` instead +of SSH, and the node end is `lnvps_node::net::apply` — the same netlink calls a +real node makes. + +```sh +sudo ./scripts/tunnel-e2e.sh # both scenarios +./scripts/tunnel-e2e.sh --filter a_guest_behind # one +``` + +Requires **root** (namespaces, veth, WireGuard) and `wireguard-tools` for the +route-server end, so the tests are `#[ignore]`d and only run from that script. + +**Why it exists.** Unit tests assert what the code *decides* — which commands +the route server issues, which netlink operations the node performs. None of +that proves a packet moves. This harness caught four things nothing else did: + +- the node's namespace was pinned from `/proc/self/ns/net`, which in a + multi-threaded process is the *process's* namespace, so every "isolated" + interface was silently landing in the operator's own network; +- WireGuard's netlink calls ran outside the namespace the interface had been + moved into, reporting "no such device" about an interface that plainly existed; +- addresses on an interface produce routes in the kernel's *local* table, which + the node then tried to delete as strays; +- **the route server never routed the pool's own block.** An address on a + point-to-point interface does not route the rest of its prefix, so a route + server holding `10.66.0.1/16` answered "network is unreachable" for every node + in the pool. Not visible in any unit test, because the code did exactly what it + was written to do. + +Coverage note: the netlink implementation (`lnvps_node::net::kernel`) and +`lnvps_node::netns` are exercised here rather than by the normal test run, the +same way `lnvps_fw`'s datapath is covered by its netns harness. Measure them +with `sudo -E cargo llvm-cov -p lnvps_node -- --include-ignored`. diff --git a/lnvps_api/src/api/marketplace.rs b/lnvps_api/src/api/marketplace.rs index ab971f5e..75387689 100644 --- a/lnvps_api/src/api/marketplace.rs +++ b/lnvps_api/src/api/marketplace.rs @@ -50,6 +50,7 @@ pub fn router() -> Router { "/api/v1/node/tunnel", get(v1_node_get_tunnel).post(v1_node_request_tunnel), ) + .route("/api/v1/node/dataplane", get(v1_node_dataplane)) } /// A node as its operator sees it. @@ -609,6 +610,63 @@ async fn v1_node_get_tunnel( ApiData::ok(allocation.into()) } +/// Everything the node's data plane should look like, in one document. +#[derive(Serialize, Debug)] +pub struct ApiNodeDataPlane { + pub tunnel: ApiNodeTunnel, + /// Gateway addresses this node must answer for on the bridge. They belong + /// to the ranges the guests were addressed from, and the guests believe + /// they are on-link. + pub gateways: Vec, + /// The guests placed here. Also the anti-spoof list: an address that is not + /// in it is not this node's to send from. + pub guests: Vec, +} + +#[derive(Serialize, Debug)] +pub struct ApiNodeGuest { + /// Host prefix (`203.0.113.5/32`), so v4 and v6 read the same way. + pub address: String, + /// The gateway this guest was configured with. + pub gateway: String, + /// The guest's MAC, when recorded. + pub mac: Option, +} + +impl From for ApiNodeDataPlane { + fn from(d: crate::provisioner::NodeDataPlane) -> Self { + Self { + gateways: d.gateways(), + guests: d + .guests + .iter() + .map(|g| ApiNodeGuest { + address: g.address.clone(), + gateway: g.gateway.clone(), + mac: g.mac.clone(), + }) + .collect(), + tunnel: d.tunnel.into(), + } + } +} + +/// The node's whole desired data plane. +/// +/// One call rather than three, because the node applies these together or not +/// at all: a bridge with no tunnel carries nothing, and a tunnel with no guest +/// routes carries nothing back. +async fn v1_node_dataplane( + auth: NodeAuth, + State(this): State, +) -> ApiResult { + let plane = crate::provisioner::node_dataplane(&this.db, &auth.node) + .await + .map_err(|e| ApiError::new(e.to_string()))? + .ok_or_else(|| ApiError::not_found("This node has no tunnel allocated yet"))?; + ApiData::ok(plane.into()) +} + /// What a node is told about itself. /// /// The daemon uses this to confirm its token works and to see whether it has diff --git a/lnvps_api/src/provisioner/tunnel.rs b/lnvps_api/src/provisioner/tunnel.rs index 46fdec88..0d37501d 100644 --- a/lnvps_api/src/provisioner/tunnel.rs +++ b/lnvps_api/src/provisioner/tunnel.rs @@ -371,6 +371,18 @@ pub async fn plan_pool(db: &Arc, pool: &TunnelPool) -> Result, pool: &TunnelPool) -> Result, +} + +/// One address assigned to a guest on a node. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GuestAddress { + /// The guest's address as a host prefix (`203.0.113.5/32`). + pub address: String, + /// The gateway the guest is configured with. + /// + /// It belongs to the range, not to this node, and the guest believes it is + /// on-link — so the node has to answer for it on the bridge. Sent per + /// address rather than per node because one node can hold guests from + /// several ranges. + pub gateway: String, + /// The guest's MAC, when one is recorded. Lets the node bind an address to + /// a port rather than trusting whatever claims it. + pub mac: Option, +} + +impl NodeDataPlane { + /// The gateway addresses this node has to answer for, deduplicated. + pub fn gateways(&self) -> Vec { + let mut out: Vec = self.guests.iter().map(|g| g.gateway.clone()).collect(); + out.sort(); + out.dedup(); + out + } +} + +/// Assemble the desired data plane for `node`. +pub async fn node_dataplane( + db: &Arc, + node: &MarketplaceNode, +) -> Result> { + let Some(tunnel) = get_node_tunnel(db, node).await? else { + return Ok(None); + }; + Ok(Some(NodeDataPlane { + guests: node_guests(db, node).await?, + tunnel, + })) +} + +/// The guests placed on `node`, with what each needs to be reachable. +pub async fn node_guests( + db: &Arc, + node: &MarketplaceNode, +) -> Result> { + let Some(host) = db.get_marketplace_node_host(node.id).await? else { + return Ok(vec![]); + }; + let mut out = Vec::new(); + for vm in db.list_vms_on_host(host.id).await? { + if vm.deleted { + continue; + } + for ip in db.list_vm_ip_assignments(vm.id).await? { + if ip.deleted { + continue; + } + let Some(address) = host_address(Some(&ip.ip)) else { + continue; + }; + // The range is what says which gateway the guest was handed; a + // node inventing one would answer for an address the guest never + // uses. Normalised to a bare address because a stored gateway may + // carry the range's prefix, and the node holds it as a host + // address on the bridge either way. + let range = db.get_ip_range(ip.ip_range_id).await?; + let gateway = lnvps_api_common::parse_gateway(&range.gateway) + .map(|g| g.ip().to_string()) + .unwrap_or(range.gateway); + out.push(GuestAddress { + address, + gateway, + mac: Some(vm.mac_address.clone()).filter(|m| !m.is_empty()), + }); + } + } + out.sort_by(|a, b| a.address.cmp(&b.address)); + Ok(out) +} + /// The public addresses assigned to the guests running on `tunnel`'s node. /// /// Empty for a tunnel that is not a marketplace node's, or a node with no @@ -959,12 +1075,27 @@ mod tests { plan.addresses, vec!["10.66.0.1/24".to_string(), "fd00:66::1/64".to_string()] ); + // ...the pool's own blocks, because an address on a point-to-point + // interface does not route the rest of its prefix, and every node in + // the pool lives in it... + assert!( + plan.routes.contains(&"10.66.0.0/24".to_string()), + "{plan:?}" + ); + assert!( + plan.routes.contains(&"fd00:66::/64".to_string()), + "{plan:?}" + ); // ...and a route for each guest address, because AllowedIPs picks the // peer for a packet already headed down the tunnel, it does not put it // there. - assert_eq!( - plan.routes, - vec!["2001:db8::5/128".to_string(), "203.0.113.5/32".to_string()] + assert!( + plan.routes.contains(&"2001:db8::5/128".to_string()), + "{plan:?}" + ); + assert!( + plan.routes.contains(&"203.0.113.5/32".to_string()), + "{plan:?}" ); assert_eq!(allocation.tunnel.pool_id, Some(pool_id)); } @@ -991,7 +1122,7 @@ mod tests { } let pool = db.get_tunnel_pool(pool_id).await.unwrap(); let plan = plan_pool(&db, &pool).await.unwrap(); - assert_eq!(plan.routes, Vec::::new()); + assert!(!plan.routes.iter().any(|r| r.starts_with("203.0.113"))); assert_eq!(plan.peers[0].allowed_ips.len(), 2, "only the node's own"); // A deleted VM takes its addressing with it for the same reason. @@ -1006,7 +1137,7 @@ mod tests { } } let plan = plan_pool(&db, &pool).await.unwrap(); - assert_eq!(plan.routes, Vec::::new()); + assert!(!plan.routes.iter().any(|r| r.starts_with("203.0.113"))); } /// A tunnel that cannot be realised contributes nothing at all, rather than @@ -1031,7 +1162,9 @@ mod tests { db.update_tunnel(&broken).await.unwrap(); let plan = plan_pool(&db, &pool).await.unwrap(); assert!(plan.peers.is_empty()); - assert!(plan.routes.is_empty()); + // The pool's blocks are routed whether or not anything is placed + // in it; the interface exists either way. + assert_eq!(plan.routes.len(), 2); // The interface still holds the pool's address: it exists whether // or not anything has been placed in it yet. assert_eq!(plan.addresses.len(), 2); @@ -1062,10 +1195,70 @@ mod tests { let plan = plan_pool(&db, &pool).await.unwrap(); assert_eq!(plan.peers.len(), 1); assert_eq!(plan.peers[0].allowed_ips, vec!["10.66.9.1/32".to_string()]); - assert!(plan.routes.is_empty()); + // The pool's own blocks, and nothing a guest brought. + assert_eq!(plan.routes.len(), 2); let _ = mock; } + /// The document is what the node acts on, so it has to carry everything the + /// node cannot work out for itself: the gateway a guest was configured with + /// belongs to the range, not to the node, and a node inventing one would + /// answer for an address no guest uses. + #[tokio::test] + async fn the_data_plane_document_describes_the_whole_node() { + let (db, mock, node, _) = fixture().await; + allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let node = db.get_marketplace_node(node.id).await.unwrap(); + let host = db + .get_marketplace_node_host(node.id) + .await + .unwrap() + .unwrap(); + add_guest(&db, &mock, host.id, &["203.0.113.5", "2001:db8::5"]).await; + + let plane = node_dataplane(&db, &node).await.unwrap().unwrap(); + assert_eq!( + plane.tunnel.tunnel.address4.as_deref(), + Some("10.66.0.2/32") + ); + assert_eq!( + plane + .guests + .iter() + .map(|g| g.address.as_str()) + .collect::>(), + ["2001:db8::5/128", "203.0.113.5/32"] + ); + // Bare, even though the range stores it with a prefix: the node holds + // it as a host address on the bridge. + assert_eq!(plane.guests[0].gateway, "10.0.0.1"); + assert_eq!(plane.guests[0].mac.as_deref(), Some("aa:bb:cc:dd:ee:ff")); + // Deduplicated: two guests from one range give the node one address to + // answer for, not two identical ones. + assert_eq!(plane.gateways(), vec!["10.0.0.1".to_string()]); + } + + /// A node with no tunnel has no data plane to describe — saying so beats + /// returning a document with an empty tunnel that a node would apply. + #[tokio::test] + async fn a_node_without_a_tunnel_has_no_document() { + let (db, _mock, node, _) = fixture().await; + assert!(node_dataplane(&db, &node).await.unwrap().is_none()); + } + + /// A node with no guests yet is still configured: it is realised before it + /// has customers, so it can be given some. + #[tokio::test] + async fn a_node_with_no_guests_still_has_a_document() { + let (db, _mock, node, _) = fixture().await; + allocate_node_tunnel(&db, &node, &NODE_KEY).await.unwrap(); + let node = db.get_marketplace_node(node.id).await.unwrap(); + + let plane = node_dataplane(&db, &node).await.unwrap().unwrap(); + assert!(plane.guests.is_empty()); + assert!(plane.gateways().is_empty()); + } + /// Give `host` a VM holding `ips`. Written through the mock's maps because /// a VM needs a template, an image and a disk that this test does not care /// about. @@ -1078,6 +1271,7 @@ mod tests { lnvps_db::Vm { id, host_id, + mac_address: "aa:bb:cc:dd:ee:ff".to_string(), ..Default::default() }, ); @@ -1087,6 +1281,7 @@ mod tests { db.insert_vm_ip_assignment(&lnvps_db::VmIpAssignment { vm_id, ip: ip.to_string(), + ip_range_id: 1, ..Default::default() }) .await diff --git a/lnvps_api/src/router/linux_ssh.rs b/lnvps_api/src/router/linux_ssh.rs index 00c8ac91..c5b0e119 100644 --- a/lnvps_api/src/router/linux_ssh.rs +++ b/lnvps_api/src/router/linux_ssh.rs @@ -28,18 +28,18 @@ pub struct LinuxSshRouter { interface: String, /// SSH private key (PEM) key: String, - /// Stands in for the SSH connection under test. + /// Where commands are run, when that is not an SSH session. /// - /// These methods run commands as root on somebody else's route server, so - /// what is worth asserting is the exact command issued — which needs the - /// transport replaced, not mocked around. - #[cfg(test)] + /// These methods run commands as root on a route server, so what is worth + /// asserting is the exact command issued — which needs the transport + /// replaced rather than mocked around. Tests record the commands; the + /// end-to-end harness runs them in a network namespace against a real + /// kernel. exec: Option, } /// A stand-in for running a command over SSH. -#[cfg(test)] -type ExecFn = std::sync::Arc OpResult + Send + Sync>; +pub type ExecFn = std::sync::Arc OpResult + Send + Sync>; impl LinuxSshRouter { /// Build a router from the stored config `url` and `token`. @@ -68,16 +68,17 @@ impl LinuxSshRouter { username, interface, key: key.to_string(), - #[cfg(test)] exec: None, }) } - /// A router whose commands are handled by `exec` instead of an SSH session. - #[cfg(test)] - fn with_exec(exec: ExecFn) -> Self { + /// A router whose commands are run by `exec` rather than over SSH. + /// + /// The transport is the only thing that changes: the commands, and the + /// decisions behind them, are the ones a real route server gets. + pub fn with_exec(exec: ExecFn) -> Self { Self { - host: "test:22".to_string(), + host: "local".to_string(), username: "root".to_string(), interface: "eth0".to_string(), key: String::new(), @@ -101,7 +102,6 @@ impl LinuxSshRouter { /// Run a command, mapping connection failures to transient errors and /// non-zero exits to fatal errors. Returns stdout on success. async fn exec_checked(&self, cmd: &str) -> OpResult { - #[cfg(test)] if let Some(exec) = &self.exec { return exec(cmd); } diff --git a/lnvps_api/src/worker.rs b/lnvps_api/src/worker.rs index 669728ac..917a7b1b 100644 --- a/lnvps_api/src/worker.rs +++ b/lnvps_api/src/worker.rs @@ -5295,11 +5295,13 @@ mod tests { vec!["10.66.0.1/24".to_string()] ); // AllowedIPs picks which peer a packet belongs to; it does not put the - // packet on the tunnel. Without this route the guest's return traffic - // is dropped as unroutable. + // packet on the tunnel. Without these routes the guest's return traffic + // is dropped as unroutable — and without the pool's own block, so is + // everything addressed to the nodes themselves, because an address on a + // point-to-point interface does not route the rest of its prefix. assert_eq!( mr.interface_routes(&interface).await, - vec!["203.0.113.5/32".to_string()] + vec!["10.66.0.0/24".to_string(), "203.0.113.5/32".to_string()] ); assert_eq!(tunnel.pool_id, Some(pool_id)); diff --git a/lnvps_e2e/Cargo.toml b/lnvps_e2e/Cargo.toml index 34d3f17f..f1d4e72b 100644 --- a/lnvps_e2e/Cargo.toml +++ b/lnvps_e2e/Cargo.toml @@ -32,6 +32,13 @@ tokio-tungstenite = "0.28" futures-util = "0.3" [dev-dependencies] +# Both ends of a marketplace tunnel, driven by the real code, in network +# namespaces: the route server's configuration path and the node daemon's. +lnvps_api = { path = "../lnvps_api" } +lnvps_api_common = { path = "../lnvps_api_common" } +lnvps_node = { path = "../lnvps_node" } +tempfile = "3" +nix = { version = "0.30", features = ["user"] } # Used only to validate the software authenticator offline: register via # webauthn-rs-core (as the server does) and verify a discoverable assertion via # the high-level webauthn-rs discoverable flow. diff --git a/lnvps_e2e/tests/tunnel_netns.rs b/lnvps_e2e/tests/tunnel_netns.rs new file mode 100644 index 00000000..e3a91c72 --- /dev/null +++ b/lnvps_e2e/tests/tunnel_netns.rs @@ -0,0 +1,437 @@ +//! Both ends of a marketplace tunnel, on a real kernel, carrying real packets. +//! +//! Everything below this line has been proved by unit tests that assert what +//! the code *decides*: which commands the route server issues, which netlink +//! operations the node performs. None of that proves a packet moves. This +//! harness builds the two ends out of network namespaces and pings across the +//! tunnel — first the node itself, then a guest sitting behind it, which is the +//! path a customer's traffic actually takes. +//! +//! ```text +//! [rs netns] [test machine's netns] [lnvps netns] [guest netns] +//! wgln <══ WireGuard ══> wgln0 created here, then ═> wgln0 veth +//! 10.66.0.1/24 its UDP socket stays here 10.66.0.2/32 br-lnvps ── 203.0.113.5/24 +//! │ 203.0.113.1/32 +//! rs_up veth ────────────────── node_up veth +//! 198.51.100.1/24 198.51.100.2/24 +//! ``` +//! +//! The shape is the production one, including the part that is easy to get +//! wrong: `wgln0` is created in the machine's own namespace so its UDP socket can +//! reach the route server through the operator's uplink, and is then moved into +//! the LNVPS namespace so that everything carried *over* the tunnel is isolated +//! from the operator's network. +//! +//! Requires root (namespaces, veths, WireGuard) so it is `#[ignore]`d; run it +//! with `scripts/tunnel-e2e.sh`. + +use std::process::Command; +use std::sync::Arc; + +use anyhow::{Context, Result, bail}; +use lnvps_api::router::{Tunnel, TunnelConfig, TunnelRouter, WireguardConfig, WireguardPeer}; +use lnvps_node::net::{DesiredDataPlane, DesiredGuest, DesiredTunnel}; + +/// Underlay: the "internet" between the two machines. +const RS_UNDERLAY: &str = "198.51.100.1/24"; +const NODE_UNDERLAY: &str = "198.51.100.2/24"; +/// Inner tunnel addresses, as the allocator hands them out. +const RS_INNER: &str = "10.66.0.1/24"; +const NODE_INNER: &str = "10.66.0.2/32"; +/// A customer address, as LNVPS assigns it to a guest on this node. +const GUEST_ADDRESS: &str = "203.0.113.5"; +const GUEST_GATEWAY: &str = "203.0.113.1"; + +/// Namespaces, torn down on drop even when a test panics. +struct Topology { + rs: String, + guest: String, + /// The node's data plane namespace, pinned where iproute2 looks so an + /// operator — and this harness — can reach it with `ip netns exec`. + dataplane: String, +} + +impl Topology { + fn new(tag: &str) -> Result { + let topology = Self { + rs: format!("lnvps-e2e-rs-{tag}"), + guest: format!("lnvps-e2e-guest-{tag}"), + dataplane: format!("lnvps-e2e-dp-{tag}"), + }; + topology.teardown(); + + run("ip", &["netns", "add", &topology.rs])?; + run("ip", &["netns", "add", &topology.guest])?; + + // The underlay: the route server and the node's machine, reachable to + // each other and to nothing else. + run( + "ip", + &[ + "link", "add", "e2e-rs", "type", "veth", "peer", "name", "e2e-node", + ], + )?; + run("ip", &["link", "set", "e2e-rs", "netns", &topology.rs])?; + topology.in_rs(&["ip", "addr", "add", RS_UNDERLAY, "dev", "e2e-rs"])?; + topology.in_rs(&["ip", "link", "set", "e2e-rs", "up"])?; + topology.in_rs(&["ip", "link", "set", "lo", "up"])?; + run("ip", &["addr", "add", NODE_UNDERLAY, "dev", "e2e-node"])?; + run("ip", &["link", "set", "e2e-node", "up"])?; + + Ok(topology) + } + + fn in_rs(&self, argv: &[&str]) -> Result { + let mut full = vec!["netns", "exec", &self.rs]; + full.extend_from_slice(argv); + run("ip", &full) + } + + fn in_guest(&self, argv: &[&str]) -> Result { + let mut full = vec!["netns", "exec", &self.guest]; + full.extend_from_slice(argv); + run("ip", &full) + } + + /// Run commands in the node's *data plane* namespace, which the production + /// code created and pinned. + fn in_dataplane(&self, argv: &[&str]) -> Result { + let mut full = vec!["netns", "exec", &self.dataplane]; + full.extend_from_slice(argv); + run("ip", &full) + } + + /// The namespace the node's code builds, as production code would. + fn open_dataplane(&self) -> Result { + lnvps_node::netns::ensure(std::path::Path::new("/run/netns"), &self.dataplane) + } + + fn teardown(&self) { + let _ = Command::new("ip") + .args(["netns", "delete", &self.rs]) + .output(); + let _ = Command::new("ip") + .args(["netns", "delete", &self.guest]) + .output(); + let _ = Command::new("ip") + .args(["netns", "delete", &self.dataplane]) + .output(); + // Anything the node's code created in the machine's namespace, which + // is where a half-finished run leaves it. + for link in ["e2e-node", "e2e-guest", "wgln0", "wg0", "br-lnvps"] { + let _ = Command::new("ip").args(["link", "del", link]).output(); + } + } +} + +impl Drop for Topology { + fn drop(&mut self) { + self.teardown(); + } +} + +fn run(program: &str, args: &[&str]) -> Result { + let out = Command::new(program) + .args(args) + .output() + .with_context(|| format!("cannot run {program}"))?; + if !out.status.success() { + bail!( + "`{program} {}` failed: {}", + args.join(" "), + String::from_utf8_lossy(&out.stderr).trim() + ); + } + Ok(String::from_utf8_lossy(&out.stdout).to_string()) +} + +/// A route server whose commands run inside a namespace instead of over SSH. +/// +/// The commands, and every decision behind them, are the ones a real route +/// server is given; only the transport changes. +fn route_server(namespace: &str) -> lnvps_api::router::LinuxSshRouter { + let namespace = namespace.to_string(); + lnvps_api::router::LinuxSshRouter::with_exec(Arc::new(move |cmd: &str| { + let out = Command::new("ip") + .args(["netns", "exec", &namespace, "sh", "-c", cmd]) + .output() + .map_err(|e| lnvps_api_common::retry::OpError::Fatal(e.into()))?; + if !out.status.success() { + return Err(lnvps_api_common::retry::OpError::Fatal(anyhow::anyhow!( + "`{cmd}` failed: {}", + String::from_utf8_lossy(&out.stderr).trim() + ))); + } + Ok(String::from_utf8_lossy(&out.stdout).to_string()) + })) +} + +/// Skip rather than fail when the machine cannot run this at all: a developer's +/// laptop without root should not report a red test it was never able to run. +fn requirements_met() -> bool { + if !nix::unistd::Uid::effective().is_root() { + eprintln!("skipping: needs root for network namespaces and WireGuard"); + return false; + } + if run("modprobe", &["wireguard"]).is_err() + && !std::path::Path::new("/sys/module/wireguard").exists() + { + eprintln!("skipping: no WireGuard support in this kernel"); + return false; + } + true +} + +/// The whole path: LNVPS's route server, the node's daemon code, and a guest +/// behind it — with packets crossing all of it. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires root and network namespaces; run with scripts/tunnel-e2e.sh"] +async fn a_guest_behind_a_node_is_reachable_from_the_route_server() -> Result<()> { + if !requirements_met() { + return Ok(()); + } + let topology = Topology::new("guest")?; + + // ---- the node generates its own key; LNVPS never sees the private half + let state_dir = tempfile::tempdir()?; + let node_key = lnvps_node::wgkey::load_or_generate(state_dir.path())?; + + // ---- the route server's own interface, configured by production code + let server_key = lnvps_api_common::generate_wireguard_keypair()?; + let rs = route_server(&topology.rs); + let interface = "wgln1"; + rs.add_tunnel(&Tunnel { + id: None, + name: interface.to_string(), + local_addr: None, + remote_addr: None, + enabled: true, + config: TunnelConfig::Wireguard(WireguardConfig { + listen_port: Some(51820), + private_key: Some(server_key.private_key.clone()), + public_key: Some(lnvps_api_common::wireguard_key_to_base64( + &server_key.public_key, + )), + peers: vec![], + }), + }) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + + // The pool's address, and the peer as the reconciler builds it: the node's + // own address plus exactly the guest addresses LNVPS assigned to it, which + // is the anti-spoof boundary. + rs.sync_tunnel_addresses(interface, &[RS_INNER.to_string()]) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + rs.set_tunnel_peer( + interface, + &WireguardPeer { + public_key: node_key.public_base64(), + endpoint: None, + allowed_ips: vec![NODE_INNER.to_string(), format!("{GUEST_ADDRESS}/32")], + persistent_keepalive: None, + }, + ) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + // The pool's block *and* the guest address: an address on a + // point-to-point interface does not route the rest of its prefix, so + // without the block the route server cannot reach any node in the pool. + rs.sync_tunnel_routes( + interface, + &["10.66.0.0/24".to_string(), format!("{GUEST_ADDRESS}/32")], + ) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + + // ---- the node applies the document LNVPS would have sent it + let kernel = lnvps_node::net::Kernel::in_namespace(topology.open_dataplane()?)?; + let desired = DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some(NODE_INNER.to_string()), + address6: None, + gateway4: Some("10.66.0.1".to_string()), + gateway6: None, + server_public_key: hex::encode(&server_key.public_key), + endpoint: "198.51.100.1:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + gateways: vec![GUEST_GATEWAY.to_string()], + guests: vec![DesiredGuest { + address: format!("{GUEST_ADDRESS}/32"), + gateway: GUEST_GATEWAY.to_string(), + mac: None, + }], + }; + lnvps_node::net::apply(&kernel, &desired, &node_key).await?; + + // ---- a guest on the node's bridge, addressed as a customer's VM is + run( + "ip", + &[ + "link", + "add", + "e2e-guest", + "type", + "veth", + "peer", + "name", + "e2e-tap", + ], + )?; + run( + "ip", + &["link", "set", "e2e-tap", "netns", &topology.dataplane], + )?; + topology.in_dataplane(&["ip", "link", "set", "e2e-tap", "master", "br-lnvps"])?; + topology.in_dataplane(&["ip", "link", "set", "e2e-tap", "up"])?; + run( + "ip", + &["link", "set", "e2e-guest", "netns", &topology.guest], + )?; + topology.in_guest(&[ + "ip", + "addr", + "add", + &format!("{GUEST_ADDRESS}/24"), + "dev", + "e2e-guest", + ])?; + topology.in_guest(&["ip", "link", "set", "e2e-guest", "up"])?; + // The guest is configured with its range's gateway and believes it is + // on-link — which is exactly why the node holds that address and answers + // for it. + topology.in_guest(&["ip", "route", "replace", "default", "via", GUEST_GATEWAY])?; + + // ---- the tunnel itself + let node_inner = NODE_INNER.split('/').next().unwrap(); + topology + .in_rs(&["ping", "-c", "3", "-W", "5", node_inner]) + .with_context(|| { + format!( + "the route server could not reach the node over the tunnel\n\ + route server:\n{}{}\n\ + node data plane:\n{}{}", + topology.in_rs(&["ip", "addr"]).unwrap_or_default(), + topology.in_rs(&["wg", "show"]).unwrap_or_default(), + topology.in_dataplane(&["ip", "addr"]).unwrap_or_default(), + topology.in_dataplane(&["wg", "show"]).unwrap_or_default(), + ) + })?; + + // ---- and the path a customer's traffic actually takes + topology + .in_rs(&["ping", "-c", "3", "-W", "5", GUEST_ADDRESS]) + .context("the route server could not reach a guest behind the node")?; + + // The node reports itself healthy only once that has happened: WireGuard comes + // up perfectly happily with a peer that never answers. + let state = lnvps_node::net::observe(&kernel).await?; + assert!(state.tunnel_up, "{state:?}"); + assert!(state.bridge_up, "{state:?}"); + assert!( + state.last_handshake_secs.is_some(), + "a tunnel that carried packets reported no handshake: {state:?}" + ); + assert!(state.healthy(), "{state:?}"); + Ok(()) +} + +/// The isolation the namespace exists for: the operator's own network is not +/// reachable from inside the data plane, and their default route is untouched. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "requires root and network namespaces; run with scripts/tunnel-e2e.sh"] +async fn the_operators_machine_keeps_its_own_network() -> Result<()> { + if !requirements_met() { + return Ok(()); + } + let topology = Topology::new("isolation")?; + let state_dir = tempfile::tempdir()?; + let node_key = lnvps_node::wgkey::load_or_generate(state_dir.path())?; + let server_key = lnvps_api_common::generate_wireguard_keypair()?; + + let default_before = run("ip", &["-j", "route", "show", "default"])?; + + // The operator's own WireGuard interface, which is why the node's is not + // called `wg0`: the node creates its interface in *this* namespace before + // moving it, so a collision here would either fail outright or, worse, + // adopt somebody's VPN and move it out from under them. + run("ip", &["link", "add", "wg0", "type", "wireguard"])?; + + let kernel = lnvps_node::net::Kernel::in_namespace(topology.open_dataplane()?)?; + lnvps_node::net::apply( + &kernel, + &DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some(NODE_INNER.to_string()), + address6: None, + gateway4: Some("10.66.0.1".to_string()), + gateway6: None, + server_public_key: hex::encode(&server_key.public_key), + endpoint: "198.51.100.1:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + gateways: vec![GUEST_GATEWAY.to_string()], + guests: vec![], + }, + &node_key, + ) + .await?; + + // The operator's interface is still theirs, still where they left it. + assert!( + run("ip", &["link", "show", "wg0"]).is_ok(), + "the operator's own wg0 was taken or destroyed" + ); + + // The interfaces exist in the namespace and nowhere else. Asserted first, + // because if this is wrong every later assertion is wrong for the same + // reason and this is the one that says why. + assert!( + run("ip", &["link", "show", "wgln0"]).is_err(), + "the node's interface is still in the machine's namespace: {}", + run("ip", &["link", "show"]).unwrap_or_default() + ); + assert!( + topology + .in_dataplane(&["ip", "link", "show", "wgln0"]) + .is_ok() + ); + assert!(run("ip", &["link", "show", "br-lnvps"]).is_err()); + + // The default route the node installed is the data plane's, not the + // machine's: taking an operator's default route would send their own + // traffic up a tunnel they do not own. + assert_eq!( + default_before, + run("ip", &["-j", "route", "show", "default"])?, + "the machine's default route changed" + ); + + // Forwarding is enabled in the namespace only. On a machine that is also + // the operator's workstation, turning it on globally is not ours to do. + let inside = topology.in_dataplane(&["cat", "/proc/sys/net/ipv4/ip_forward"]); + assert_eq!(inside.unwrap_or_default().trim(), "1"); + + assert!( + topology + .in_dataplane(&["ip", "link", "show", "e2e-node"]) + .is_err(), + "the operator's uplink is reachable from inside the data plane" + ); + Ok(()) +} + +/// LNVPS and the node hold the bridge name as a constant each. They are not +/// sent to each other precisely so they cannot disagree at runtime — which only +/// works if they agree at build time, and this is where both are in scope. +#[test] +fn both_ends_agree_on_the_bridge() { + assert_eq!( + lnvps_api::provisioner::NODE_BRIDGE, + lnvps_node::net::GUEST_BRIDGE + ); +} diff --git a/lnvps_node/Cargo.toml b/lnvps_node/Cargo.toml index a9710e54..428d8c67 100644 --- a/lnvps_node/Cargo.toml +++ b/lnvps_node/Cargo.toml @@ -15,6 +15,15 @@ path = "src/main.rs" lnvps_host_util = { path = "../lnvps_host_util", default-features = false } nostr = { version = "0.44", default-features = false, features = ["std"] } +# The node's own WireGuard key, generated here so LNVPS never holds the private +# half. Generated in-process rather than by shelling out to `wg genkey`, and +# without depending on lnvps_api_common, which would pull the database, axum and +# the payment stack onto somebody else's hardware. +x25519-dalek = { version = "3.0.0", features = ["static_secrets"] } +rand = "0.9" +hex.workspace = true +# Outbound calls to LNVPS: the node fetches the data plane it should be running. +reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] } # Self-signed TLS identity, pinned by LNVPS at registration. Same crate and # approach as lnvps_fw's control API. rcgen = "0.13" @@ -38,9 +47,18 @@ env_logger.workspace = true serde.workspace = true serde_json.workspace = true config.workspace = true +rtnetlink = "0.21" +defguard_wireguard_rs = "0.11" +futures-util = "0.3.33" +async-trait.workspace = true +ipnetwork.workspace = true +netlink-packet-route = "0.30" +nix = { version = "0.30", features = ["sched", "mount", "fs"] } [dev-dependencies] tempfile = "3" +# Builds responses for the API-client tests without a live server. +http = "1" # Drives the router directly in tests, so authentication is exercised without # binding a socket or terminating TLS. tower = { version = "0.5", features = ["util"] } diff --git a/lnvps_node/config.example.yaml b/lnvps_node/config.example.yaml index ceba0c0c..29116241 100644 --- a/lnvps_node/config.example.yaml +++ b/lnvps_node/config.example.yaml @@ -25,6 +25,11 @@ heartbeat-secs: 60 # Inbound control API. Omit this section until the node is paired — without it # the daemon has nothing to serve and will say so. +# +# The daemon brings the tunnel up itself before binding, so `listen` is an +# address that exists because of what the daemon did, not something to configure +# by hand: it is the address LNVPS allocated, which `lnvps-node dataplane show` +# prints. control: # Must be an address of tunnel-interface below, and never a wildcard. The # control API can start and stop other people's virtual machines, so it is @@ -32,4 +37,7 @@ control: # real addresses at startup. listen: "10.66.0.2" port: 8890 - tunnel-interface: "wg0" + # Named with the LNVPS prefix rather than wg0: the daemon creates it on the + # machine before moving it into its own network namespace, and an operator's + # own wg0 is a name it must not collide with. + tunnel-interface: "wgln0" diff --git a/lnvps_node/src/api.rs b/lnvps_node/src/api.rs new file mode 100644 index 00000000..819f4d1a --- /dev/null +++ b/lnvps_node/src/api.rs @@ -0,0 +1,278 @@ +//! Outbound calls to LNVPS. +//! +//! This is the only direction that works before a tunnel exists, which is why +//! the node fetches its data plane rather than being handed it: the control API +//! LNVPS would push it over is reachable *through* the tunnel it describes. +//! +//! Every call carries the node's own token. A node authenticates as itself, +//! never as its operator, so a compromised machine costs the operator that node +//! and nothing else. + +use anyhow::{Context, Result, bail}; +use serde::Deserialize; + +use crate::credential::Credential; +use crate::net::DesiredDataPlane; + +/// A client for the LNVPS node API. +pub struct LnvpsApi { + base_url: String, + authorization: String, + http: reqwest::Client, +} + +/// The envelope every LNVPS response comes in. +#[derive(Deserialize)] +struct ApiResponse { + data: Option, + error: Option, +} + +impl LnvpsApi { + pub fn new(base_url: &str, credential: &Credential) -> Result { + Ok(Self { + base_url: base_url.trim_end_matches('/').to_string(), + authorization: credential.authorization_header(), + http: reqwest::Client::builder() + // A node that cannot reach LNVPS must find out in seconds. The + // default is no timeout at all, which turns an unreachable API + // into a daemon that never finishes starting. + .timeout(std::time::Duration::from_secs(30)) + .build() + .context("Cannot build an HTTP client")?, + }) + } + + /// Present the node's public key and receive its tunnel allocation. + /// + /// Idempotent at the far end: a node that asks twice gets the allocation it + /// already has, and one presenting a new key is re-pinned in place. + pub async fn request_tunnel(&self, public_key: &[u8; 32]) -> Result<()> { + let url = format!("{}/api/v1/node/tunnel", self.base_url); + let response = self + .http + .post(&url) + .header("Authorization", &self.authorization) + .json(&serde_json::json!({ "public_key": hex::encode(public_key) })) + .send() + .await + .with_context(|| format!("Cannot reach LNVPS at {url}"))?; + let _: serde_json::Value = decode(response, &url).await?; + Ok(()) + } + + /// Fetch the data plane this node should be running. + pub async fn dataplane(&self) -> Result { + let url = format!("{}/api/v1/node/dataplane", self.base_url); + let response = self + .http + .get(&url) + .header("Authorization", &self.authorization) + .send() + .await + .with_context(|| format!("Cannot reach LNVPS at {url}"))?; + decode(response, &url).await + } +} + +/// Unwrap an LNVPS response, preferring its own error message to the status. +/// +/// The API reports failure in the body as often as in the status, and "500 +/// Internal Server Error" tells an operator nothing they can act on where +/// "This node has no tunnel allocated yet" tells them exactly what to do. +async fn decode( + response: reqwest::Response, + url: &str, +) -> Result { + let status = response.status(); + let body = response + .text() + .await + .with_context(|| format!("Cannot read the response from {url}"))?; + + match serde_json::from_str::>(&body) { + Ok(parsed) => { + if let Some(error) = parsed.error { + bail!("{url}: {error}"); + } + match parsed.data { + Some(data) => Ok(data), + None => bail!("{url}: response carried neither data nor an error"), + } + } + // Not an LNVPS envelope at all: a proxy error page, or the wrong URL. + // Reporting the status and a bounded slice of the body beats a serde + // message about a missing field. + Err(e) => bail!( + "{url}: {status}, and the response is not an LNVPS response ({e}): {}", + body.chars().take(200).collect::() + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Mutex; + + #[derive(Deserialize, Debug, PartialEq)] + struct Thing { + value: u32, + } + + fn response(status: u16, body: &str) -> reqwest::Response { + reqwest::Response::from( + http::Response::builder() + .status(status) + .body(body.to_string()) + .unwrap(), + ) + } + + /// The ordinary case: the envelope's data is what the caller wanted. + #[tokio::test] + async fn a_successful_response_is_unwrapped() { + let got: Thing = decode(response(200, r#"{"data":{"value":7}}"#), "u") + .await + .unwrap(); + assert_eq!(got, Thing { value: 7 }); + } + + /// The API reports failure in the body as often as in the status, and its + /// message is the one an operator can act on. + #[tokio::test] + async fn the_apis_own_error_is_preferred_to_the_status() { + let err = decode::( + response(404, r#"{"error":"This node has no tunnel allocated yet"}"#), + "u", + ) + .await + .unwrap_err(); + assert!(format!("{err}").contains("no tunnel allocated"), "{err}"); + } + + /// A proxy error page is not an LNVPS response; saying so beats a serde + /// message about a missing field. + #[tokio::test] + async fn a_non_lnvps_response_is_reported_with_its_status() { + let err = decode::(response(502, "bad gateway"), "u") + .await + .unwrap_err(); + let text = format!("{err}"); + assert!(text.contains("502"), "{text}"); + assert!(text.contains("bad gateway"), "{text}"); + } + + /// An envelope carrying neither half is a bug at the far end, and a node + /// that treated it as success would apply an empty data plane. + #[tokio::test] + async fn an_empty_envelope_is_an_error() { + assert!(decode::(response(200, "{}"), "u").await.is_err()); + } + + fn credential() -> Credential { + Credential::parse( + "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuaWQiOjd9.c2ln", + std::path::Path::new("/etc/lnvps-node/token"), + ) + .unwrap() + } + + /// A real server on a real socket, because what is worth proving is the + /// request that goes out: the node's token on every call, the public key + /// presented before the document is asked for, and the document parsed as + /// the node will actually apply it. + #[tokio::test] + async fn the_node_presents_its_key_and_fetches_its_document() { + use axum::routing::{get, post}; + use std::sync::Arc; + + let seen: Arc>> = Arc::new(Mutex::new(Vec::new())); + let recorder = seen.clone(); + let app = axum::Router::new() + .route( + "/api/v1/node/tunnel", + post(|headers: axum::http::HeaderMap, body: String| async move { + recorder.lock().unwrap().push(( + "POST /api/v1/node/tunnel".to_string(), + headers + .get("authorization") + .and_then(|h| h.to_str().ok()) + .unwrap_or_default() + .to_string(), + body, + )); + axum::Json(serde_json::json!({ "data": {} })) + }), + ) + .route( + "/api/v1/node/dataplane", + get(|| async { + axum::Json(serde_json::json!({ "data": { + "tunnel": { + "address4": "10.66.0.2/32", + "address6": null, + "gateway4": "10.66.0.1", + "gateway6": null, + "server_public_key": "ab".repeat(32), + "endpoint": "rs1.example:51820", + "keepalive": 25, + "mtu": 1420 + }, + "gateways": ["203.0.113.1"], + "guests": [ + {"address": "203.0.113.5/32", "gateway": "203.0.113.1", "mac": null} + ] + }})) + }), + ); + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); + + let api = LnvpsApi::new(&format!("http://{addr}"), &credential()).unwrap(); + api.request_tunnel(&[0x11; 32]).await.unwrap(); + let plane = api.dataplane().await.unwrap(); + + let calls = seen.lock().unwrap().clone(); + assert_eq!(calls.len(), 1); + assert!(calls[0].1.starts_with("Bearer "), "{:?}", calls[0]); + // Hex, as the rest of the node API states keys. + assert!( + calls[0].2.contains(&hex::encode([0x11u8; 32])), + "{:?}", + calls[0] + ); + + assert_eq!(plane.tunnel.mtu, 1420); + assert_eq!(plane.guests.len(), 1); + assert_eq!(plane.gateways, vec!["203.0.113.1".to_string()]); + } + + /// An unreachable API is a message naming the URL, not a bare connection + /// error: an operator reading a node's log has to know what it was dialling. + #[tokio::test] + async fn an_unreachable_api_names_what_it_was_dialling() { + // Port 1 on localhost: nothing listens there, and nothing can. + let api = LnvpsApi::new("http://127.0.0.1:1", &credential()).unwrap(); + let err = api.dataplane().await.unwrap_err(); + assert!(format!("{err:#}").contains("127.0.0.1:1"), "{err:#}"); + let err = api.request_tunnel(&[0x11; 32]).await.unwrap_err(); + assert!(format!("{err:#}").contains("node/tunnel"), "{err:#}"); + } + + /// The base URL is normalised so a trailing slash in the config file does + /// not produce `//api/v1/...`, which some proxies redirect and others 404. + #[test] + fn the_base_url_is_normalised() { + let credential = Credential::parse( + "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuaWQiOjd9.c2ln", + std::path::Path::new("/etc/lnvps-node/token"), + ) + .unwrap(); + let api = LnvpsApi::new("https://api.lnvps.net/", &credential).unwrap(); + assert_eq!(api.base_url, "https://api.lnvps.net"); + assert!(api.authorization.starts_with("Bearer ")); + } +} diff --git a/lnvps_node/src/config.rs b/lnvps_node/src/config.rs index 1264fafd..12b0688f 100644 --- a/lnvps_node/src/config.rs +++ b/lnvps_node/src/config.rs @@ -80,7 +80,7 @@ fn default_control_port() -> u16 { } fn default_tunnel_interface() -> String { - "wg0".to_string() + crate::net::TUNNEL_INTERFACE.to_string() } impl NodeConfig { @@ -257,7 +257,7 @@ control: let control = NodeConfig::load(&path).unwrap().control.unwrap(); assert_eq!(control.listen, v4("10.66.0.1")); assert_eq!(control.port, 8890); - assert_eq!(control.tunnel_interface, "wg0"); + assert_eq!(control.tunnel_interface, "wgln0"); } /// A typo in a key must not be silently ignored: a misspelled `listen` diff --git a/lnvps_node/src/control.rs b/lnvps_node/src/control.rs index 3029513b..f80fa1a9 100644 --- a/lnvps_node/src/control.rs +++ b/lnvps_node/src/control.rs @@ -57,17 +57,32 @@ pub struct ControlState { /// is compared against this, so a request signed for one node cannot be /// replayed against another by setting `Host` to the first node's address. pub base_url: String, + /// How the node reads its own network back. Held here so a test can supply + /// a machine that is not the one the tests run on. + pub net: Arc, } impl ControlState { /// Build state for a node serving on `addr`. - pub fn new(control_pubkey: PublicKey, addr: SocketAddr) -> Self { + pub fn new( + control_pubkey: PublicKey, + addr: SocketAddr, + net: Arc, + ) -> Self { Self { control_pubkey, replay: Mutex::new(ReplayGuard::new(REPLAY_WINDOW, REPLAY_CAPACITY)), base_url: format!("https://{addr}"), + net, } } + + /// The data plane as this machine actually has it. + pub async fn observe(&self) -> crate::net::DataPlaneState { + crate::net::observe(self.net.as_ref()) + .await + .unwrap_or_default() + } } /// What the node reports about itself. @@ -77,6 +92,11 @@ pub struct NodeStatus { pub version: &'static str, /// Host inventory, the same view `lnvps-node inventory` prints. pub inventory: crate::inventory::Inventory, + /// The data plane as the machine actually has it — read from the machine, + /// never remembered from what was applied. This is the first thing the + /// health gate checks, so a node that quietly lost its tunnel has to be + /// able to say so. + pub dataplane: crate::net::DataPlaneState, } /// The control router, with authentication layered over everything. @@ -92,6 +112,22 @@ pub fn router(state: Arc) -> Router { /// Serve the control API over HTTPS until the process exits. pub async fn serve(state: Arc, addr: SocketAddr, tls: NodeTls) -> Result<()> { + let listener = std::net::TcpListener::bind(addr) + .with_context(|| format!("Cannot bind the control API to {addr}"))?; + serve_on(state, listener, tls).await +} + +/// Serve on a socket somebody else opened. +/// +/// The tunnel address lives in the data plane's network namespace, so the +/// listening socket has to be created in there — a socket keeps the namespace +/// it was created in, which is what lets the rest of the daemon stay in the +/// machine's own namespace and go on reaching LNVPS. +pub async fn serve_on( + state: Arc, + listener: std::net::TcpListener, + tls: NodeTls, +) -> Result<()> { // rustls needs a process-wide crypto provider; installing twice is not an // error worth failing a startup over. let _ = rustls::crypto::ring::default_provider().install_default(); @@ -100,11 +136,17 @@ pub async fn serve(state: Arc, addr: SocketAddr, tls: NodeTls) -> .await .context("Control API TLS configuration is invalid")?; + let addr = listener + .local_addr() + .context("The control API socket has no address")?; log::info!( "Control API listening on https://{addr} (certificate fingerprint {})", tls.fingerprint ); - axum_server::bind_rustls(addr, cfg) + listener + .set_nonblocking(true) + .context("Cannot make the control API socket non-blocking")?; + axum_server::from_tcp_rustls(listener, cfg) .serve(router(state).into_make_service()) .await .context("Control API server stopped") @@ -177,10 +219,18 @@ fn unauthorized(reason: &str) -> Response { } /// Report what this node is and what it is running. -async fn get_status() -> Json { +async fn get_status(State(state): State>) -> Json { Json(NodeStatus { version: env!("CARGO_PKG_VERSION"), inventory: crate::inventory::Inventory::collect(), + // Observed on demand rather than cached: a cached answer is a report + // that the tunnel was up once, which is exactly the thing the gate must + // not accept. + // Observed on demand rather than cached: a cached answer is a report + // that the tunnel was up once, which is exactly what the health gate + // must not accept. A machine this cannot be read from reports nothing + // configured, which is a truthful answer and one the gate can act on. + dataplane: state.observe().await, }) } @@ -196,7 +246,11 @@ mod tests { const ADDR: &str = "10.66.0.7:8890"; fn state_for(keys: &Keys, addr: &str) -> Arc { - Arc::new(ControlState::new(keys.public_key(), addr.parse().unwrap())) + Arc::new(ControlState::new( + keys.public_key(), + addr.parse().unwrap(), + Arc::new(crate::net::tests::FakeKernel::default()), + )) } /// A NIP-98 header, with every field overridable so tests can tamper with @@ -244,6 +298,25 @@ mod tests { assert_eq!(code, StatusCode::OK); } + /// The status report is servable on a machine with no data plane at all — + /// the state a node is in before it has been configured, and one the health + /// gate has to be able to read rather than time out on. + #[tokio::test] + async fn the_status_is_served_before_there_is_a_data_plane() { + let keys = Keys::generate(); + let addr: SocketAddr = ADDR.parse().unwrap(); + let state = ControlState::new( + keys.public_key(), + addr, + Arc::new(crate::net::tests::FakeKernel::default()), + ); + + let url = format!("https://{ADDR}/api/v1/status"); + let auth = auth_header(&keys, &url, "GET", b""); + let code = status_of(Arc::new(state), get("/api/v1/status", Some(&auth))).await; + assert_eq!(code, StatusCode::OK); + } + #[tokio::test] async fn an_unsigned_request_is_refused() { let keys = Keys::generate(); diff --git a/lnvps_node/src/lib.rs b/lnvps_node/src/lib.rs index bb0308ea..94b0e074 100644 --- a/lnvps_node/src/lib.rs +++ b/lnvps_node/src/lib.rs @@ -12,11 +12,21 @@ //! - [`tls`] — the node's TLS identity, whose fingerprint LNVPS pins at //! registration, so the node's *replies* are authenticated too. //! - [`inventory`] — what the node reports about the machine. +//! - [`api`] — outbound calls to LNVPS, the only direction that works before +//! there is a tunnel. +//! - [`net`] — applying the data plane LNVPS asked for, over netlink. +//! - [`netns`] — the namespace that data plane lives in, so LNVPS configures +//! its own network rather than the operator's. +//! - [`wgkey`] — the node's WireGuard key, generated here and never sent. //! - [`config`] — configuration, including where the control API may listen. +pub mod api; pub mod config; pub mod control; pub mod control_auth; pub mod credential; pub mod inventory; +pub mod net; +pub mod netns; pub mod tls; +pub mod wgkey; diff --git a/lnvps_node/src/main.rs b/lnvps_node/src/main.rs index 9a180738..e18f0d7b 100644 --- a/lnvps_node/src/main.rs +++ b/lnvps_node/src/main.rs @@ -3,6 +3,7 @@ use std::net::SocketAddr; use std::path::{Path, PathBuf}; use std::sync::Arc; +use std::time::Duration; use anyhow::{Context, Result}; use clap::{Parser, Subcommand}; @@ -31,6 +32,11 @@ enum Command { Inventory, /// Check the configuration and credential without contacting LNVPS. Check, + /// Show or apply the data plane LNVPS has asked this node to run. + Dataplane { + #[command(subcommand)] + action: DataplaneAction, + }, /// Print the node's TLS fingerprint, the value LNVPS pins at registration. Fingerprint { /// State directory holding the identity (defaults to the configured one). @@ -39,6 +45,16 @@ enum Command { }, } +#[derive(Subcommand)] +enum DataplaneAction { + /// Fetch what LNVPS wants and print it, changing nothing. + Show, + /// Fetch it and apply it. + Apply, + /// Print what this machine currently has, read from the machine itself. + Observe, +} + #[tokio::main] async fn main() -> Result<()> { env_logger::init(); @@ -73,6 +89,7 @@ async fn main() -> Result<()> { None => println!("control: not configured (node not yet paired)"), } } + Command::Dataplane { action } => dataplane(&cli.config, action).await?, Command::Fingerprint { state_dir } => { let state_dir = match state_dir { Some(dir) => dir, @@ -91,6 +108,49 @@ async fn main() -> Result<()> { Ok(()) } +/// Show, apply or observe the data plane. +/// +/// Exposed as a command of its own so an operator can see exactly what LNVPS +/// asked for, and re-drive it, without running the daemon or reading its logs. +async fn dataplane(config_path: &Path, action: DataplaneAction) -> Result<()> { + let config = NodeConfig::load(config_path)?; + + // Deliberately before the credential is loaded: "what does this machine + // have?" is the question an operator asks when something is wrong, and it + // must not fail because the token is missing or LNVPS is unreachable. + if let DataplaneAction::Observe = action { + let state = lnvps_node::net::observe(&lnvps_node::net::Kernel::new()?).await?; + println!("{}", serde_json::to_string_pretty(&state)?); + return Ok(()); + } + + let credential = Credential::load_checked(&config.credential)?; + let api = lnvps_node::api::LnvpsApi::new(&config.api_url, &credential)?; + + // Presented before fetching: the document describes a tunnel that does not + // exist until LNVPS has this node's public key, so asking for it first + // would report "no tunnel allocated" on every first run. + let key = lnvps_node::wgkey::load_or_generate(&config.state_dir)?; + if key.generated { + log::info!("Generated this node's tunnel key; presenting it to LNVPS"); + } + api.request_tunnel(&key.public_bytes()).await?; + let desired = api.dataplane().await?; + + match action { + DataplaneAction::Show => println!("{}", serde_json::to_string_pretty(&desired)?), + DataplaneAction::Apply => { + let kernel = lnvps_node::net::Kernel::new()?; + let applied = lnvps_node::net::apply(&kernel, &desired, &key).await?; + for line in applied { + println!("{line}"); + } + } + DataplaneAction::Observe => unreachable!("handled above, before loading a credential"), + } + Ok(()) +} + /// Start the control API. /// /// The order here is deliberate: everything that can be known to be wrong is @@ -108,9 +168,28 @@ async fn run(config_path: &Path) -> Result<()> { // than starting a listener that can never authorise anything (decision 12). let control_pubkey = control_auth::control_pubkey()?; + // The data plane is applied *before* the listen address is checked, because + // the address being checked for is one this brings into existence: the + // control API binds the tunnel interface, and on a fresh machine the tunnel + // does not exist until now. + let kernel = Arc::new(lnvps_node::net::Kernel::new()?); + if let Err(e) = apply_dataplane(&config, kernel.as_ref()).await { + // Not fatal. A node whose tunnel is already up from a previous run must + // keep serving through an LNVPS outage — refusing to start would turn + // an API blip into every node on the platform going dark. + log::warn!( + "Could not apply the data plane ({e}); continuing with whatever this machine \ + already has configured" + ); + } + // Decision 13: the address must belong to the tunnel interface, checked - // against the interface itself. - let addrs = config::interface_addresses(&control.tunnel_interface)?; + // against the interface itself — inside the data plane namespace, which is + // the only place that interface exists. + let interface = control.tunnel_interface.clone(); + let addrs = kernel + .namespace() + .enter(move || config::interface_addresses(&interface))?; config::validate_listen_address(control.listen, &addrs)?; let tls = tls::load_or_generate(&config.state_dir, Some(control.listen))?; @@ -124,5 +203,56 @@ async fn run(config_path: &Path) -> Result<()> { } let addr = SocketAddr::new(control.listen, control.port); - control::serve(Arc::new(ControlState::new(control_pubkey, addr)), addr, tls).await + + // Re-applied on a timer for the same reason the route server reconciles its + // end: guests come and go, and a node that only configured itself at + // startup would route a departed customer's address until it was restarted. + let refresh = config.clone(); + let refresh_kernel = kernel.clone(); + tokio::spawn(async move { + let interval = Duration::from_secs(refresh.heartbeat_secs.max(10)); + loop { + tokio::time::sleep(interval).await; + if let Err(e) = apply_dataplane(&refresh, refresh_kernel.as_ref()).await { + log::warn!("Data plane refresh failed: {e}"); + } + } + }); + + // Bound inside the namespace, because that is where the tunnel address + // lives. The socket keeps that namespace while the rest of the daemon stays + // in the machine's own, which is what lets it go on reaching LNVPS. + let listener = kernel.namespace().enter(move || { + std::net::TcpListener::bind(addr) + .with_context(|| format!("Cannot bind the control API to {addr}")) + })?; + + control::serve_on( + Arc::new(ControlState::new(control_pubkey, addr, kernel)), + listener, + tls, + ) + .await +} + +/// Fetch the data plane and apply it. +async fn apply_dataplane(config: &NodeConfig, kernel: &dyn lnvps_node::net::NetOps) -> Result<()> { + let credential = Credential::load_checked(&config.credential)?; + let api = lnvps_node::api::LnvpsApi::new(&config.api_url, &credential)?; + + let key = lnvps_node::wgkey::load_or_generate(&config.state_dir)?; + if key.generated { + log::warn!( + "Generated a new tunnel key. LNVPS re-pins a node that presents one, but the \ + tunnel stays down until it has been presented." + ); + } + api.request_tunnel(&key.public_bytes()).await?; + let desired = api.dataplane().await?; + + let applied = lnvps_node::net::apply(kernel, &desired, &key).await?; + if !applied.is_empty() { + log::debug!("Applied data plane: {}", applied.join("; ")); + } + Ok(()) } diff --git a/lnvps_node/src/net.rs b/lnvps_node/src/net.rs new file mode 100644 index 00000000..7ebdcaad --- /dev/null +++ b/lnvps_node/src/net.rs @@ -0,0 +1,1012 @@ +//! Applying the data plane LNVPS asked for. +//! +//! The daemon configures the machine itself rather than writing files for +//! something else to read. A marketplace node runs on hardware LNVPS does not +//! own: a data plane that depends on the operator having wired it up correctly +//! is one whose mistakes surface as a customer's VM having no network. +//! +//! Interfaces, addresses and routes are managed over **netlink**, not by +//! shelling out to `ip`. Netlink is the interface the kernel actually offers; +//! `ip` is a program that formats netlink messages and then formats the answer +//! back into text for us to parse. Going direct means no dependency on +//! iproute2's presence or version, no output parsing that changes between +//! releases, no arguments to quote, and errors that arrive as kernel error +//! codes instead of a line of English on stderr. +//! +//! Everything is stated declaratively and converges: a node that is already +//! right is not disturbed, and one that has drifted is corrected without being +//! torn down. +//! +//! The kernel calls sit behind [`NetOps`] so the orchestration can be tested +//! without root: what is worth asserting is *what the node decides to do*. +//! Whether the netlink implementation of those decisions really works on a +//! kernel is proven by the netns end-to-end harness, which runs both ends of a +//! real tunnel and pings across it. + +use std::collections::HashSet; +use std::net::IpAddr; +use std::path::Path; + +use anyhow::{Context, Result, bail}; +use async_trait::async_trait; +use ipnetwork::IpNetwork; +use serde::{Deserialize, Serialize}; + +use crate::wgkey::{self, NodeKey}; + +/// The tunnel interface the node terminates its data plane on. +/// +/// `wgln0`, not `wg0`: the interface is created in the *machine's* namespace +/// before being moved into the data plane's, and an operator's own `wg0` — a +/// VPN, a mesh, anything — is a name we would collide with there. The `wgln` +/// prefix is the same one LNVPS uses for its route-server interfaces, so a +/// managed interface is recognisable as ours wherever it turns up. +pub const TUNNEL_INTERFACE: &str = "wgln0"; + +/// The bridge guests sit on. +/// +/// A constant here rather than a field in the data-plane document, because the +/// daemon needs the name before it has ever spoken to LNVPS: `dataplane +/// observe` reports on it without a credential, and an operator debugging a +/// node asks about it offline. A document that could name a different bridge +/// would leave the node holding two answers to one question. LNVPS holds the +/// same constant, and the end-to-end harness asserts the two agree. +pub const GUEST_BRIDGE: &str = "br-lnvps"; + +/// The desired data plane, as LNVPS states it. +/// +/// Mirrors `GET /api/v1/node/dataplane`. Fetched and applied as one document +/// because it only makes sense as one: a bridge with no tunnel carries nothing, +/// and a tunnel with no guest routes carries nothing back. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct DesiredDataPlane { + pub tunnel: DesiredTunnel, + /// Gateway addresses this node answers for on the bridge. + #[serde(default)] + pub gateways: Vec, + #[serde(default)] + pub guests: Vec, +} + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct DesiredTunnel { + pub address4: Option, + pub address6: Option, + pub gateway4: Option, + pub gateway6: Option, + /// The route server's key, hex. + pub server_public_key: String, + pub endpoint: String, + pub keepalive: Option, + pub mtu: u16, +} + +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq)] +pub struct DesiredGuest { + pub address: String, + pub gateway: String, + pub mac: Option, +} + +/// How a WireGuard interface should be configured. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WgSettings { + /// Base64, as WireGuard states keys. + pub private_key: String, + pub peer_public_key: String, + pub endpoint: String, + pub keepalive: Option, + /// Everything, for a node: its guests use LNVPS addresses, so no traffic of + /// theirs belongs anywhere else. + pub allowed_ips: Vec, +} + +/// What a WireGuard interface currently is. +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct WgObserved { + /// Peers keyed by public key, with seconds since each last handshake. + pub peers: Vec<(String, Option)>, +} + +/// The kernel operations the data plane needs. +/// +/// A trait so the orchestration above it can be tested without root, and so the +/// end-to-end harness can drive the same code inside a network namespace. +#[async_trait] +pub trait NetOps: Send + Sync { + /// Whether a link exists. + async fn link_exists(&self, name: &str) -> Result; + /// Create a WireGuard interface. + async fn create_wireguard(&self, name: &str) -> Result<()>; + /// Create a bridge. + async fn create_bridge(&self, name: &str) -> Result<()>; + /// Bring a link up with the given MTU. + async fn set_up(&self, name: &str, mtu: u32) -> Result<()>; + /// Whether a link is up, and its MTU. + async fn link_state(&self, name: &str) -> Result<(bool, Option)>; + + async fn addresses(&self, name: &str) -> Result>; + async fn add_address(&self, name: &str, address: IpNetwork) -> Result<()>; + async fn del_address(&self, name: &str, address: IpNetwork) -> Result<()>; + + /// Destinations routed out of `name`. + async fn routes(&self, name: &str) -> Result>; + async fn add_route(&self, destination: IpNetwork, name: &str) -> Result<()>; + async fn del_route(&self, destination: IpNetwork, name: &str) -> Result<()>; + + /// Configure the WireGuard interface: key, peer, allowed IPs. + async fn configure_wireguard(&self, name: &str, settings: &WgSettings) -> Result<()>; + /// Remove a peer by public key. + async fn remove_wireguard_peer(&self, name: &str, public_key: &str) -> Result<()>; + /// Read the interface back, or `None` when it does not exist. + async fn wireguard_state(&self, name: &str) -> Result>; + + /// Read a kernel knob, or `None` when this kernel does not have it. + async fn sysctl(&self, key: &str) -> Result>; + async fn set_sysctl(&self, key: &str, value: &str) -> Result<()>; +} + +/// What the machine currently looks like. +/// +/// Reported to LNVPS over the control API, where it is the first thing the +/// health gate checks. Every field is read from the machine, never remembered +/// from what was applied — the point of observing is to catch the case where +/// the two disagree. +#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] +pub struct DataPlaneState { + pub tunnel_up: bool, + /// Seconds since the last handshake with the route server. `None` when + /// there has never been one, which is the difference between "configured" + /// and "working". + pub last_handshake_secs: Option, + pub tunnel_mtu: Option, + pub bridge_up: bool, + pub forwarding4: bool, + pub forwarding6: bool, + /// Guest addresses actually routed to the bridge. + pub routed_guests: usize, +} + +impl DataPlaneState { + /// Whether this node can carry a customer. + /// + /// A handshake is required, not just an interface: WireGuard comes up happily + /// with a peer that never answers, and a node in that state looks + /// configured while being unreachable. + pub fn healthy(&self) -> bool { + self.tunnel_up && self.last_handshake_secs.is_some() && self.bridge_up && self.forwarding4 + } +} + +/// Apply `desired` to this machine. +/// +/// Returns a description of what was changed, so `dataplane apply` can show an +/// operator what happened and a test can assert it. An empty list means the +/// machine was already right, which is the normal case on every refresh after +/// the first. +pub async fn apply( + ops: &dyn NetOps, + desired: &DesiredDataPlane, + key: &NodeKey, +) -> Result> { + let mut changed = Vec::new(); + apply_tunnel(ops, desired, key, &mut changed).await?; + apply_bridge(ops, desired, &mut changed).await?; + apply_forwarding(ops, &mut changed).await?; + Ok(changed) +} + +/// Bring up the tunnel interface and point the default route down it. +async fn apply_tunnel( + ops: &dyn NetOps, + desired: &DesiredDataPlane, + key: &NodeKey, + changed: &mut Vec, +) -> Result<()> { + if !ops.link_exists(TUNNEL_INTERFACE).await? { + ops.create_wireguard(TUNNEL_INTERFACE).await?; + changed.push(format!("created {TUNNEL_INTERFACE}")); + } + + let peer_key = wgkey::parse_public_key(&desired.tunnel.server_public_key)?; + let settings = WgSettings { + private_key: key.private_base64(), + peer_public_key: peer_key.clone(), + endpoint: desired.tunnel.endpoint.clone(), + keepalive: desired.tunnel.keepalive, + // Everything goes up the tunnel: the node's guests use LNVPS addresses, + // so there is no traffic of theirs that belongs anywhere else. + allowed_ips: vec![ + "0.0.0.0/0".parse().expect("a constant CIDR"), + "::/0".parse().expect("a constant CIDR"), + ], + }; + ops.configure_wireguard(TUNNEL_INTERFACE, &settings).await?; + changed.push(format!("configured {TUNNEL_INTERFACE}")); + + // A peer that is not the route server has no business on this interface. + // It would most likely be a stale key from a re-key, still able to send + // traffic that the node treats as coming from LNVPS. + if let Some(observed) = ops.wireguard_state(TUNNEL_INTERFACE).await? { + for (stale, _) in observed.peers.iter().filter(|(k, _)| *k != peer_key) { + ops.remove_wireguard_peer(TUNNEL_INTERFACE, stale).await?; + changed.push(format!("removed stale peer {stale}")); + } + } + + let want = tunnel_addresses(desired)?; + sync_addresses(ops, TUNNEL_INTERFACE, &want, changed).await?; + + // Not 1500: WireGuard's overhead comes off it, and guessing wrong hangs + // large transfers rather than failing outright. + ops.set_up(TUNNEL_INTERFACE, desired.tunnel.mtu as u32) + .await?; + + // No gateway: the tunnel is point-to-point, so the interface names the next + // hop by itself, and naming one would be a second copy of the route + // server's address free to disagree with the peer's. + let existing = ops.routes(TUNNEL_INTERFACE).await?; + for default in default_routes(desired) { + if !existing.contains(&default) { + ops.add_route(default, TUNNEL_INTERFACE).await?; + changed.push(format!("routed {default} via {TUNNEL_INTERFACE}")); + } + } + Ok(()) +} + +/// Bring up the guest bridge and route each guest to it. +async fn apply_bridge( + ops: &dyn NetOps, + desired: &DesiredDataPlane, + changed: &mut Vec, +) -> Result<()> { + if !ops.link_exists(GUEST_BRIDGE).await? { + ops.create_bridge(GUEST_BRIDGE).await?; + changed.push(format!("created {GUEST_BRIDGE}")); + } + // The bridge carries the same payload as the tunnel, so it takes the same + // MTU: a guest that sends 1500 bytes into a 1420-byte tunnel produces a + // connection that opens and then hangs on the first large transfer. + ops.set_up(GUEST_BRIDGE, desired.tunnel.mtu as u32).await?; + + // The gateway belongs to the range, not to this node, and the guest + // believes it is on-link. Held as a host address so the node answers for it + // without claiming the rest of the range is local — the other addresses in + // it live on other nodes, up the tunnel. + let mut want = Vec::new(); + for gateway in &desired.gateways { + want.push(host_prefix(gateway)?); + } + sync_addresses(ops, GUEST_BRIDGE, &want, changed).await?; + + // The guest thinks its neighbours are on-link and will ARP for them; proxy + // ARP is what lets the node answer and pull that traffic up the tunnel + // instead of it disappearing into a link that has no such address. + for knob in [ + format!("net/ipv4/conf/{GUEST_BRIDGE}/proxy_arp"), + format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp"), + ] { + set_if_needed(ops, &knob, "1", changed).await?; + } + + let mut guests: HashSet = HashSet::new(); + for guest in &desired.guests { + guests.insert(host_prefix(&guest.address)?); + } + let existing: HashSet = ops.routes(GUEST_BRIDGE).await?.into_iter().collect(); + + let mut to_add: Vec<&IpNetwork> = guests.difference(&existing).collect(); + to_add.sort(); + for address in to_add { + ops.add_route(*address, GUEST_BRIDGE).await?; + changed.push(format!("routed {address} to {GUEST_BRIDGE}")); + } + + // A guest that has been deleted or moved must stop being routed here at + // once: its address goes back in the pool and may already be somebody + // else's. Routes the kernel maintains for the link itself are left alone — + // deleting the IPv6 link-local prefix to tidy a list would take the + // interface's own connectivity with it. + let mut to_drop: Vec<&IpNetwork> = existing + .difference(&guests) + .filter(|r| !is_link_local(r)) + .collect(); + to_drop.sort(); + for address in to_drop { + ops.del_route(*address, GUEST_BRIDGE).await?; + changed.push(format!("unrouted {address} from {GUEST_BRIDGE}")); + } + Ok(()) +} + +/// A node that does not forward is a node whose guests have no network at all. +async fn apply_forwarding(ops: &dyn NetOps, changed: &mut Vec) -> Result<()> { + for knob in ["net/ipv4/ip_forward", "net/ipv6/conf/all/forwarding"] { + set_if_needed(ops, knob, "1", changed).await?; + } + Ok(()) +} + +/// Make the addresses on `name` exactly `want`. +async fn sync_addresses( + ops: &dyn NetOps, + name: &str, + want: &[IpNetwork], + changed: &mut Vec, +) -> Result<()> { + let existing = ops.addresses(name).await?; + for address in want { + if !existing.contains(address) { + ops.add_address(name, *address).await?; + changed.push(format!("added {address} to {name}")); + } + } + for address in &existing { + // A link-local address is the kernel's, not ours. Removing it would + // remove the interface's ability to talk to itself, on every refresh. + if is_link_local(address) || want.contains(address) { + continue; + } + ops.del_address(name, *address).await?; + changed.push(format!("removed {address} from {name}")); + } + Ok(()) +} + +/// Write a kernel knob only when it does not already say what it should. +/// +/// A knob the kernel does not have is skipped rather than fatal: IPv6 can be +/// compiled out, and a node with no IPv6 guests is still a working node. +async fn set_if_needed( + ops: &dyn NetOps, + key: &str, + value: &str, + changed: &mut Vec, +) -> Result<()> { + match ops.sysctl(key).await? { + Some(current) if current.trim() == value => Ok(()), + Some(_) => { + ops.set_sysctl(key, value).await?; + changed.push(format!("set {key}={value}")); + Ok(()) + } + None => Ok(()), + } +} + +/// The addresses the tunnel interface should carry. +fn tunnel_addresses(desired: &DesiredDataPlane) -> Result> { + [&desired.tunnel.address4, &desired.tunnel.address6] + .into_iter() + .flatten() + .map(|a| { + a.parse::() + .with_context(|| format!("LNVPS sent {a}, which is not an address")) + }) + .collect() +} + +/// The default routes to install, one per family the tunnel has an address in. +/// +/// A family with no address gets no default route: it would black-hole that +/// family's traffic rather than leaving the machine's own routing to handle it. +fn default_routes(desired: &DesiredDataPlane) -> Vec { + let mut out = Vec::new(); + if desired.tunnel.address4.is_some() { + out.push("0.0.0.0/0".parse().expect("a constant CIDR")); + } + if desired.tunnel.address6.is_some() { + out.push("::/0".parse().expect("a constant CIDR")); + } + out +} + +/// `203.0.113.1` -> `203.0.113.1/32`, and the v6 equivalent. A value that +/// already carries a prefix is taken as it is. +fn host_prefix(address: &str) -> Result { + if address.contains('/') { + return address + .parse::() + .with_context(|| format!("{address} is not a CIDR")); + } + let ip: IpAddr = address + .parse() + .with_context(|| format!("{address} is not an IP address"))?; + Ok(IpNetwork::from(ip)) +} + +/// Addresses the kernel manages for itself. +fn is_link_local(address: &IpNetwork) -> bool { + match address.ip() { + IpAddr::V4(v4) => v4.is_link_local() || v4.is_loopback(), + IpAddr::V6(v6) => (v6.segments()[0] & 0xffc0) == 0xfe80 || v6.is_loopback(), + } +} + +/// Read back what the machine actually has. +pub async fn observe(ops: &dyn NetOps) -> Result { + let (tunnel_up, tunnel_mtu) = ops.link_state(TUNNEL_INTERFACE).await?; + let (bridge_up, _) = ops.link_state(GUEST_BRIDGE).await?; + let last_handshake_secs = ops + .wireguard_state(TUNNEL_INTERFACE) + .await? + .and_then(|w| w.peers.into_iter().filter_map(|(_, age)| age).min()); + Ok(DataPlaneState { + tunnel_up, + tunnel_mtu, + last_handshake_secs, + bridge_up, + forwarding4: enabled(ops, "net/ipv4/ip_forward").await?, + forwarding6: enabled(ops, "net/ipv6/conf/all/forwarding").await?, + routed_guests: ops.routes(GUEST_BRIDGE).await?.len(), + }) +} + +async fn enabled(ops: &dyn NetOps, key: &str) -> Result { + Ok(ops + .sysctl(key) + .await? + .map(|v| v.trim() == "1") + .unwrap_or(false)) +} + +/// Where the kernel exposes its knobs. A path rather than the `sysctl` binary: +/// one less program a node has to have installed, and a write that either +/// happens or reports why. +const PROC_SYS: &str = "/proc/sys"; + +/// Read a kernel knob from `/proc/sys`. +pub fn read_sysctl(root: &Path, key: &str) -> Result> { + let path = root.join(key); + match std::fs::read_to_string(&path) { + Ok(value) => Ok(Some(value)), + // Absent means this kernel does not have the knob — IPv6 can be + // compiled out — which is a fact about the machine, not a failure. + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e).with_context(|| format!("Cannot read {}", path.display())), + } +} + +/// Write a kernel knob to `/proc/sys`. +pub fn write_sysctl(root: &Path, key: &str, value: &str) -> Result<()> { + let path = root.join(key); + std::fs::write(&path, value).with_context(|| format!("Cannot set {}", path.display())) +} + +pub use kernel::Kernel; + +/// A machine whose network cannot be read. +/// +/// Not a mock: it is what a node that has never been configured looks like from +/// outside, and reporting that truthfully is what lets the health gate say +/// "this node has no data plane" instead of timing out. +pub struct UnavailableKernel; + +#[async_trait] +impl NetOps for UnavailableKernel { + async fn link_exists(&self, _name: &str) -> Result { + Ok(false) + } + async fn create_wireguard(&self, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn create_bridge(&self, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn set_up(&self, _name: &str, _mtu: u32) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn link_state(&self, _name: &str) -> Result<(bool, Option)> { + Ok((false, None)) + } + async fn addresses(&self, _name: &str) -> Result> { + Ok(vec![]) + } + async fn add_address(&self, _name: &str, _address: IpNetwork) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn del_address(&self, _name: &str, _address: IpNetwork) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn routes(&self, _name: &str) -> Result> { + Ok(vec![]) + } + async fn add_route(&self, _destination: IpNetwork, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn del_route(&self, _destination: IpNetwork, _name: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn configure_wireguard(&self, _name: &str, _settings: &WgSettings) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn remove_wireguard_peer(&self, _name: &str, _public_key: &str) -> Result<()> { + bail!("This node cannot configure its network") + } + async fn wireguard_state(&self, _name: &str) -> Result> { + Ok(None) + } + async fn sysctl(&self, _key: &str) -> Result> { + Ok(None) + } + async fn set_sysctl(&self, _key: &str, _value: &str) -> Result<()> { + bail!("This node cannot configure its network") + } +} + +/// The real implementation: netlink for links, addresses and routes, the +/// kernel's WireGuard netlink interface for the tunnel, and `/proc/sys` for the +/// forwarding knobs. +mod kernel { + use super::*; + + use defguard_wireguard_rs::key::Key; + use defguard_wireguard_rs::net::IpAddrMask; + use defguard_wireguard_rs::peer::Peer; + use defguard_wireguard_rs::{ + InterfaceConfiguration, Kernel as WgKernel, WGApi, WireguardInterfaceApi, + }; + use futures_util::TryStreamExt; + use netlink_packet_route::address::AddressAttribute; + use netlink_packet_route::link::LinkFlags; + use netlink_packet_route::route::{RouteAddress, RouteAttribute, RouteHeader}; + use rtnetlink::{Handle, LinkBridge, LinkUnspec, RouteMessageBuilder, new_connection}; + + /// Talks to the kernel, inside the data plane's own network namespace. + /// + /// The namespace is why this type exists at all: a marketplace node is + /// often not only a marketplace node, and configuring routes, forwarding + /// and proxy ARP in the machine's own namespace would be configuring the + /// operator's network for them. See [`crate::netns`]. + pub struct Kernel { + handle: Handle, + namespace: crate::netns::Handle, + } + + impl Kernel { + /// Open a netlink connection inside the data plane namespace, + /// creating the namespace if this is the first run. + pub fn new() -> Result { + Self::in_namespace(crate::netns::ensure_default()?) + } + + /// Same, for an already-open namespace. Used by the end-to-end harness, + /// which builds its own. + pub fn in_namespace(namespace: crate::netns::Handle) -> Result { + // The socket is opened *inside* the namespace: a netlink socket + // belongs to the namespace it was created in, so everything sent + // over this one lands there no matter which thread sends it. + // + // The runtime handle goes with it because the socket registers with + // tokio's reactor as it is created, and the thread that enters the + // namespace is a bare one — without this it panics with "there is + // no reactor running", which is a confusing way to discover that a + // namespace and a runtime are different kinds of context. + let runtime = tokio::runtime::Handle::current(); + let (connection, handle, _) = namespace.enter(move || { + let _guard = runtime.enter(); + new_connection().context("Cannot open a netlink socket") + })?; + tokio::spawn(connection); + Ok(Self { handle, namespace }) + } + + /// The namespace this configures. + pub fn namespace(&self) -> &crate::netns::Handle { + &self.namespace + } + + async fn index(&self, name: &str) -> Result> { + let mut links = self + .handle + .link() + .get() + .match_name(name.to_string()) + .execute(); + match links.try_next().await { + Ok(Some(link)) => Ok(Some(link.header.index)), + // "no such device" arrives as an error, and it is an answer + // rather than a fault: the caller is asking whether to create it. + Ok(None) | Err(_) => Ok(None), + } + } + + async fn require_index(&self, name: &str) -> Result { + self.index(name) + .await? + .with_context(|| format!("Interface {name} does not exist")) + } + } + + #[async_trait] + impl NetOps for Kernel { + async fn link_exists(&self, name: &str) -> Result { + Ok(self.index(name).await?.is_some()) + } + + async fn create_wireguard(&self, name: &str) -> Result<()> { + // Deliberately created in the machine's own namespace and then + // moved: a WireGuard interface keeps its UDP socket in the + // namespace it was created in. Created inside, it could only reach + // a route server through itself. This way the encrypted outer + // traffic still leaves by the operator's uplink while everything + // carried over the tunnel stays isolated. + let mut api = WGApi::::new(name.to_string()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + api.create_interface() + .with_context(|| format!("Cannot create WireGuard interface {name}"))?; + self.move_into_namespace(name).await + } + + async fn create_bridge(&self, name: &str) -> Result<()> { + // Created inside: nothing about a bridge needs the machine's own + // namespace, and a guest port must never be attachable from there. + self.handle + .link() + .add(LinkBridge::new(name).build()) + .execute() + .await + .with_context(|| format!("Cannot create bridge {name}")) + } + + async fn set_up(&self, name: &str, mtu: u32) -> Result<()> { + let index = self.require_index(name).await?; + self.handle + .link() + .set(LinkUnspec::new_with_index(index).up().mtu(mtu).build()) + .execute() + .await + .with_context(|| format!("Cannot bring {name} up with MTU {mtu}")) + } + + async fn link_state(&self, name: &str) -> Result<(bool, Option)> { + let mut links = self + .handle + .link() + .get() + .match_name(name.to_string()) + .execute(); + let Ok(Some(link)) = links.try_next().await else { + return Ok((false, None)); + }; + let up = link.header.flags.contains(LinkFlags::Up); + let mtu = link.attributes.iter().find_map(|a| { + if let netlink_packet_route::link::LinkAttribute::Mtu(mtu) = a { + Some(*mtu) + } else { + None + } + }); + Ok((up, mtu)) + } + + async fn addresses(&self, name: &str) -> Result> { + let Some(index) = self.index(name).await? else { + return Ok(vec![]); + }; + let mut addresses = self + .handle + .address() + .get() + .set_link_index_filter(index) + .execute(); + let mut out = Vec::new(); + while let Some(message) = addresses.try_next().await? { + let prefix = message.header.prefix_len; + for attribute in message.attributes { + if let AddressAttribute::Address(ip) = attribute + && let Ok(network) = IpNetwork::new(ip, prefix) + { + out.push(network); + } + } + } + Ok(out) + } + + async fn add_address(&self, name: &str, address: IpNetwork) -> Result<()> { + let index = self.require_index(name).await?; + self.handle + .address() + .add(index, address.ip(), address.prefix()) + .execute() + .await + .with_context(|| format!("Cannot add {address} to {name}")) + } + + async fn del_address(&self, name: &str, address: IpNetwork) -> Result<()> { + let index = self.require_index(name).await?; + let mut addresses = self + .handle + .address() + .get() + .set_link_index_filter(index) + .execute(); + while let Some(message) = addresses.try_next().await? { + let matches = message.header.prefix_len == address.prefix() + && message + .attributes + .iter() + .any(|a| matches!(a, AddressAttribute::Address(ip) if *ip == address.ip())); + if matches { + return self + .handle + .address() + .del(message) + .execute() + .await + .with_context(|| format!("Cannot remove {address} from {name}")); + } + } + Ok(()) + } + + async fn routes(&self, name: &str) -> Result> { + let Some(index) = self.index(name).await? else { + return Ok(vec![]); + }; + let mut out = Vec::new(); + // Both families asked for separately, because netlink dumps one + // family at a time: a v6 guest would otherwise look unrouted on + // every pass and be re-added forever. + for builder in [ + RouteMessageBuilder::::new().build(), + RouteMessageBuilder::::new().build(), + ] { + let mut routes = self.handle.route().get(builder).execute(); + while let Some(route) = routes.try_next().await? { + let on_this_link = route + .attributes + .iter() + .any(|a| matches!(a, RouteAttribute::Oif(oif) if *oif == index)); + // Only the main table. Giving an interface an address makes + // the kernel write entries into the *local* table for it, + // and treating those as ours means trying to delete the + // bridge's own gateway on every sweep — which fails, and + // rightly so. + if !on_this_link || route.header.table != RouteHeader::RT_TABLE_MAIN { + continue; + } + let destination = route.attributes.iter().find_map(|a| match a { + RouteAttribute::Destination(RouteAddress::Inet(v4)) => { + Some(IpAddr::from(*v4)) + } + RouteAttribute::Destination(RouteAddress::Inet6(v6)) => { + Some(IpAddr::from(*v6)) + } + _ => None, + }); + let prefix = route.header.destination_prefix_length; + let network = match destination { + Some(ip) => IpNetwork::new(ip, prefix).ok(), + // No destination at all is the default route. + None if route.header.address_family + == netlink_packet_route::AddressFamily::Inet => + { + "0.0.0.0/0".parse().ok() + } + None => "::/0".parse().ok(), + }; + if let Some(network) = network { + out.push(network); + } + } + } + Ok(out) + } + + async fn add_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + let index = self.require_index(name).await?; + let message = match destination { + IpNetwork::V4(v4) => RouteMessageBuilder::::new() + .destination_prefix(v4.ip(), v4.prefix()) + .output_interface(index) + .build(), + IpNetwork::V6(v6) => RouteMessageBuilder::::new() + .destination_prefix(v6.ip(), v6.prefix()) + .output_interface(index) + .build(), + }; + self.handle + .route() + .add(message) + .replace() + .execute() + .await + .with_context(|| format!("Cannot route {destination} via {name}")) + } + + async fn del_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + let index = self.require_index(name).await?; + let message = match destination { + IpNetwork::V4(v4) => RouteMessageBuilder::::new() + .destination_prefix(v4.ip(), v4.prefix()) + .output_interface(index) + .build(), + IpNetwork::V6(v6) => RouteMessageBuilder::::new() + .destination_prefix(v6.ip(), v6.prefix()) + .output_interface(index) + .build(), + }; + self.handle + .route() + .del(message) + .execute() + .await + .with_context(|| format!("Cannot remove the route for {destination} on {name}")) + } + + async fn configure_wireguard(&self, name: &str, settings: &WgSettings) -> Result<()> { + // Inside the namespace: the interface was moved there, and + // WireGuard's netlink socket, like every other, belongs to the + // namespace of the thread that opens it. Configuring from outside + // reports "no such device" about an interface that plainly exists. + let (name, settings) = (name.to_string(), settings.clone()); + self.namespace.enter(move || configure(&name, &settings)) + } + + async fn remove_wireguard_peer(&self, name: &str, public_key: &str) -> Result<()> { + let (name, public_key) = (name.to_string(), public_key.to_string()); + self.namespace.enter(move || { + let api = WGApi::::new(name.clone()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + let key = Key::from_str(&public_key).context("Not a WireGuard key")?; + api.remove_peer(&key) + .with_context(|| format!("Cannot remove peer {public_key} from {name}")) + }) + } + + async fn wireguard_state(&self, name: &str) -> Result> { + if !self.link_exists(name).await? { + return Ok(None); + } + let name = name.to_string(); + self.namespace + .enter(move || observe_wireguard(&name)) + .map(Some) + } + + async fn sysctl(&self, key: &str) -> Result> { + // `/proc/sys/net` reflects the reading thread's namespace, so this + // has to be read from inside — otherwise the node would report the + // operator's forwarding setting as its own. + let key = key.to_string(); + self.namespace + .enter(move || read_sysctl(Path::new(PROC_SYS), &key)) + } + + async fn set_sysctl(&self, key: &str, value: &str) -> Result<()> { + let (key, value) = (key.to_string(), value.to_string()); + self.namespace + .enter(move || write_sysctl(Path::new(PROC_SYS), &key, &value)) + } + } + + /// Configure a WireGuard interface. Runs on a thread already inside the + /// data plane namespace. + fn configure(name: &str, settings: &WgSettings) -> Result<()> { + let api = WGApi::::new(name.to_string()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + let peer = Peer { + public_key: Key::from_str(&settings.peer_public_key) + .context("The route server's key is not a WireGuard key")?, + endpoint: Some( + resolve(&settings.endpoint) + .with_context(|| format!("Cannot resolve endpoint {}", settings.endpoint))?, + ), + persistent_keepalive_interval: settings.keepalive, + allowed_ips: settings + .allowed_ips + .iter() + .map(|n| IpAddrMask::new(n.ip(), n.prefix())) + .collect(), + ..Default::default() + }; + // Addresses and MTU are handled through netlink above rather than + // here, so that one code path owns them for both interfaces. + let config = InterfaceConfiguration { + name: name.to_string(), + prvkey: settings.private_key.clone(), + addresses: vec![], + port: 0, + peers: vec![peer], + mtu: None, + fwmark: None, + }; + api.configure_interface(&config) + .with_context(|| format!("Cannot configure WireGuard interface {name}")) + } + + /// Read a WireGuard interface back. Runs inside the namespace, as above. + fn observe_wireguard(name: &str) -> Result { + let api = WGApi::::new(name.to_string()) + .with_context(|| format!("Cannot address WireGuard interface {name}"))?; + let host = api + .read_interface_data() + .with_context(|| format!("Cannot read WireGuard interface {name}"))?; + let now = std::time::SystemTime::now(); + Ok(WgObserved { + peers: host + .peers + .values() + .map(|p| { + let age = p + .last_handshake + // The zero time means "never", not 1970: reporting + // an age of half a century would look like a stale + // tunnel rather than one that has never worked. + .filter(|t| *t != std::time::UNIX_EPOCH) + .and_then(|t| now.duration_since(t).ok()) + .map(|d| d.as_secs()); + (p.public_key.to_string(), age) + }) + .collect(), + }) + } + + impl Kernel { + /// Move an interface from the machine's namespace into the data + /// plane's. + /// + /// Uses a second netlink socket, opened in the machine's own namespace, + /// because the move is asked of the namespace the interface is *in*. + async fn move_into_namespace(&self, name: &str) -> Result<()> { + let (connection, handle, _) = + new_connection().context("Cannot open a netlink socket")?; + tokio::spawn(connection); + + let mut links = handle.link().get().match_name(name.to_string()).execute(); + let Ok(Some(link)) = links.try_next().await else { + // Already inside: `create_wireguard` is the only caller, and a + // retry after a partial failure finds it where it belongs. + return Ok(()); + }; + handle + .link() + .set( + LinkUnspec::new_with_index(link.header.index) + .setns_by_fd(self.namespace.as_raw_fd()) + .build(), + ) + .execute() + .await + .with_context(|| format!("Cannot move {name} into the data plane namespace")) + } + } + + /// Resolve `host:port`, which is what a node is told to dial. + /// + /// Resolved here rather than passed through as text because the kernel + /// takes an address: a name that does not resolve has to fail with that + /// message, not as a rejected netlink attribute. + fn resolve(endpoint: &str) -> Result { + use std::net::ToSocketAddrs; + endpoint + .to_socket_addrs()? + .next() + .with_context(|| format!("{endpoint} resolved to no addresses")) + } + + trait KeyFromStr: Sized { + fn from_str(value: &str) -> Result; + } + + impl KeyFromStr for Key { + /// WireGuard states keys in base64; the crate wants bytes. + fn from_str(value: &str) -> Result { + use base64::Engine; + let raw = base64::engine::general_purpose::STANDARD + .decode(value.trim()) + .context("A WireGuard key must be base64")?; + let bytes: [u8; 32] = raw + .as_slice() + .try_into() + .map_err(|_| anyhow::anyhow!("A WireGuard key is 32 bytes, got {}", raw.len()))?; + Ok(Key::new(bytes)) + } + } +} + +#[cfg(test)] +pub mod tests; diff --git a/lnvps_node/src/net/tests.rs b/lnvps_node/src/net/tests.rs new file mode 100644 index 00000000..dc68f194 --- /dev/null +++ b/lnvps_node/src/net/tests.rs @@ -0,0 +1,509 @@ +//! Tests for the orchestration: what the node *decides* to do to a machine. +//! +//! The kernel sits behind [`NetOps`], so these run without root and assert the +//! decisions rather than the netlink encoding of them. Whether those decisions +//! really work on a kernel is proven by the end-to-end harness, which builds +//! both ends of a real tunnel in network namespaces and pings across it. + +use std::collections::HashMap; +use std::sync::Mutex; + +use super::*; + +/// A machine that remembers what it was told, and answers questions from what +/// it has been told so far — so a second `apply` sees the first one's work, +/// which is what makes "converges and then goes quiet" testable. +#[derive(Default)] +pub struct FakeKernel { + links: Mutex)>>, + addresses: Mutex>>, + routes: Mutex>>, + wireguard: Mutex>, + settings: Mutex>, + sysctls: Mutex>, + /// Knobs this kernel does not have at all. + missing: Vec, +} + +impl FakeKernel { + fn new() -> Self { + let sysctls = HashMap::from([ + ("net/ipv4/ip_forward".to_string(), "0".to_string()), + ("net/ipv6/conf/all/forwarding".to_string(), "0".to_string()), + ( + format!("net/ipv4/conf/{GUEST_BRIDGE}/proxy_arp"), + "0".to_string(), + ), + ( + format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp"), + "0".to_string(), + ), + ]); + Self { + sysctls: Mutex::new(sysctls), + ..Default::default() + } + } + + fn addresses_of(&self, name: &str) -> Vec { + self.addresses + .lock() + .unwrap() + .get(name) + .cloned() + .unwrap_or_default() + } + + fn routes_of(&self, name: &str) -> Vec { + self.routes + .lock() + .unwrap() + .get(name) + .cloned() + .unwrap_or_default() + } + + fn wg_settings(&self) -> Option { + self.settings.lock().unwrap().clone() + } + + fn sysctl_value(&self, key: &str) -> Option { + self.sysctls.lock().unwrap().get(key).cloned() + } +} + +#[async_trait] +impl NetOps for FakeKernel { + async fn link_exists(&self, name: &str) -> Result { + Ok(self.links.lock().unwrap().contains_key(name)) + } + + async fn create_wireguard(&self, name: &str) -> Result<()> { + self.links + .lock() + .unwrap() + .insert(name.to_string(), (false, None)); + self.wireguard + .lock() + .unwrap() + .insert(name.to_string(), WgObserved::default()); + Ok(()) + } + + async fn create_bridge(&self, name: &str) -> Result<()> { + self.links + .lock() + .unwrap() + .insert(name.to_string(), (false, None)); + Ok(()) + } + + async fn set_up(&self, name: &str, mtu: u32) -> Result<()> { + self.links + .lock() + .unwrap() + .insert(name.to_string(), (true, Some(mtu))); + Ok(()) + } + + async fn link_state(&self, name: &str) -> Result<(bool, Option)> { + Ok(self + .links + .lock() + .unwrap() + .get(name) + .copied() + .unwrap_or((false, None))) + } + + async fn addresses(&self, name: &str) -> Result> { + Ok(self.addresses_of(name)) + } + + async fn add_address(&self, name: &str, address: IpNetwork) -> Result<()> { + self.addresses + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .push(address); + Ok(()) + } + + async fn del_address(&self, name: &str, address: IpNetwork) -> Result<()> { + if let Some(list) = self.addresses.lock().unwrap().get_mut(name) { + list.retain(|a| *a != address); + } + Ok(()) + } + + async fn routes(&self, name: &str) -> Result> { + Ok(self.routes_of(name)) + } + + async fn add_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + self.routes + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .push(destination); + Ok(()) + } + + async fn del_route(&self, destination: IpNetwork, name: &str) -> Result<()> { + if let Some(list) = self.routes.lock().unwrap().get_mut(name) { + list.retain(|r| *r != destination); + } + Ok(()) + } + + async fn configure_wireguard(&self, name: &str, settings: &WgSettings) -> Result<()> { + *self.settings.lock().unwrap() = Some(settings.clone()); + self.wireguard + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .peers + .retain(|(k, _)| *k != settings.peer_public_key); + self.wireguard + .lock() + .unwrap() + .entry(name.to_string()) + .or_default() + .peers + .push((settings.peer_public_key.clone(), None)); + Ok(()) + } + + async fn remove_wireguard_peer(&self, name: &str, public_key: &str) -> Result<()> { + if let Some(state) = self.wireguard.lock().unwrap().get_mut(name) { + state.peers.retain(|(k, _)| k != public_key); + } + Ok(()) + } + + async fn wireguard_state(&self, name: &str) -> Result> { + Ok(self.wireguard.lock().unwrap().get(name).cloned()) + } + + async fn sysctl(&self, key: &str) -> Result> { + if self.missing.iter().any(|m| m == key) { + return Ok(None); + } + Ok(self.sysctl_value(key)) + } + + async fn set_sysctl(&self, key: &str, value: &str) -> Result<()> { + self.sysctls + .lock() + .unwrap() + .insert(key.to_string(), value.to_string()); + Ok(()) + } +} + +fn desired() -> DesiredDataPlane { + DesiredDataPlane { + tunnel: DesiredTunnel { + address4: Some("10.66.0.2/32".to_string()), + address6: Some("fd00:66::2/128".to_string()), + gateway4: Some("10.66.0.1".to_string()), + gateway6: Some("fd00:66::1".to_string()), + server_public_key: hex::encode([0xab; 32]), + endpoint: "rs1.example:51820".to_string(), + keepalive: Some(25), + mtu: 1420, + }, + gateways: vec!["203.0.113.1".to_string()], + guests: vec![DesiredGuest { + address: "203.0.113.5/32".to_string(), + gateway: "203.0.113.1".to_string(), + mac: Some("aa:bb:cc:dd:ee:ff".to_string()), + }], + } +} + +fn key() -> NodeKey { + let dir = tempfile::tempdir().unwrap(); + wgkey::load_or_generate(dir.path()).unwrap() +} + +fn cidr(value: &str) -> IpNetwork { + value.parse().unwrap() +} + +/// A machine with nothing configured must end up with the whole data plane. +/// Any one part missing is a customer with no network. +#[tokio::test] +async fn a_bare_machine_gets_the_whole_data_plane() { + let kernel = FakeKernel::new(); + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + assert!(!changed.is_empty()); + + assert_eq!( + kernel.addresses_of(TUNNEL_INTERFACE), + vec![cidr("10.66.0.2/32"), cidr("fd00:66::2/128")] + ); + assert_eq!( + kernel.link_state(TUNNEL_INTERFACE).await.unwrap(), + (true, Some(1420)) + ); + // The bridge carries the same payload, so it takes the same MTU: a guest + // sending 1500 bytes into a 1420-byte tunnel opens a connection and then + // hangs on the first large transfer. + assert_eq!( + kernel.link_state(GUEST_BRIDGE).await.unwrap(), + (true, Some(1420)) + ); + + // Everything goes up the tunnel: the guests use LNVPS addresses, so no + // traffic of theirs belongs anywhere else. + let settings = kernel.wg_settings().unwrap(); + assert_eq!(settings.allowed_ips, vec![cidr("0.0.0.0/0"), cidr("::/0")]); + assert_eq!(settings.keepalive, Some(25)); + assert_eq!(settings.endpoint, "rs1.example:51820"); + + let mut tunnel_routes = kernel.routes_of(TUNNEL_INTERFACE); + tunnel_routes.sort(); + assert_eq!(tunnel_routes, vec![cidr("0.0.0.0/0"), cidr("::/0")]); + + // The gateway belongs to the range, not the node, and is held as a host + // address so the node answers for it without claiming the rest of the + // range is local. + assert_eq!( + kernel.addresses_of(GUEST_BRIDGE), + vec![cidr("203.0.113.1/32")] + ); + assert_eq!(kernel.routes_of(GUEST_BRIDGE), vec![cidr("203.0.113.5/32")]); + + // The guest thinks its neighbours are on-link and will ARP for them; proxy + // ARP is what lets the node answer and pull that traffic up the tunnel. + assert_eq!( + kernel.sysctl_value(&format!("net/ipv4/conf/{GUEST_BRIDGE}/proxy_arp")), + Some("1".to_string()) + ); + assert_eq!( + kernel.sysctl_value(&format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp")), + Some("1".to_string()) + ); + assert_eq!( + kernel.sysctl_value("net/ipv4/ip_forward"), + Some("1".to_string()) + ); + assert_eq!( + kernel.sysctl_value("net/ipv6/conf/all/forwarding"), + Some("1".to_string()) + ); +} + +/// A machine that is already right must be left alone. This runs every minute +/// on hardware LNVPS does not own; churn there is a tunnel that flaps. +#[tokio::test] +async fn a_correct_machine_is_not_touched_again() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + // Configuring WireGuard is stated unconditionally — the kernel API takes + // the whole interface and there is nothing to compare a private key + // against — but nothing else may move. + assert_eq!(changed, vec![format!("configured {TUNNEL_INTERFACE}")]); + assert_eq!(kernel.addresses_of(TUNNEL_INTERFACE).len(), 2); + assert_eq!(kernel.routes_of(GUEST_BRIDGE), vec![cidr("203.0.113.5/32")]); +} + +/// A peer that is not the route server has no business on this interface — +/// most likely a stale key from a re-key, still able to send traffic the node +/// would treat as LNVPS's. +#[tokio::test] +async fn a_stale_peer_is_removed() { + let kernel = FakeKernel::new(); + kernel.create_wireguard(TUNNEL_INTERFACE).await.unwrap(); + kernel + .wireguard + .lock() + .unwrap() + .get_mut(TUNNEL_INTERFACE) + .unwrap() + .peers + .push(("c3RyYXk=".to_string(), None)); + + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + assert!( + changed + .iter() + .any(|c| c.contains("removed stale peer c3RyYXk=")), + "{changed:?}" + ); + let peers = kernel + .wireguard_state(TUNNEL_INTERFACE) + .await + .unwrap() + .unwrap() + .peers; + assert_eq!(peers.len(), 1); +} + +/// A guest that has been deleted or moved must stop being routed here at once: +/// its address goes back in the pool and may already be somebody else's. +#[tokio::test] +async fn a_departed_guest_stops_being_routed() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + kernel + .add_route(cidr("203.0.113.9/32"), GUEST_BRIDGE) + .await + .unwrap(); + + let changed = apply(&kernel, &desired(), &key()).await.unwrap(); + assert!( + changed + .iter() + .any(|c| c.contains("unrouted 203.0.113.9/32")), + "{changed:?}" + ); + assert_eq!(kernel.routes_of(GUEST_BRIDGE), vec![cidr("203.0.113.5/32")]); + // The bridge's own gateway is an address, not a route, so a sweep of + // departed guests cannot take the bridge's addressing with it. + assert_eq!( + kernel.addresses_of(GUEST_BRIDGE), + vec![cidr("203.0.113.1/32")] + ); +} + +/// An address that is no longer ours goes, but the kernel's own link-local +/// address stays: removing it would break the interface, on every refresh. +#[tokio::test] +async fn a_stale_address_goes_and_the_kernels_own_stays() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + kernel + .add_address(TUNNEL_INTERFACE, cidr("10.66.0.9/32")) + .await + .unwrap(); + kernel + .add_address(TUNNEL_INTERFACE, cidr("fe80::1/64")) + .await + .unwrap(); + + apply(&kernel, &desired(), &key()).await.unwrap(); + let addresses = kernel.addresses_of(TUNNEL_INTERFACE); + assert!(!addresses.contains(&cidr("10.66.0.9/32")), "{addresses:?}"); + assert!(addresses.contains(&cidr("fe80::1/64")), "{addresses:?}"); +} + +/// A single-stack pool must not produce a default route for the family it has +/// no address in: that route black-holes traffic instead of leaving the +/// machine's own routing to handle it. +#[tokio::test] +async fn a_single_stack_tunnel_only_routes_its_own_family() { + let kernel = FakeKernel::new(); + let mut plane = desired(); + plane.tunnel.address6 = None; + apply(&kernel, &plane, &key()).await.unwrap(); + assert_eq!(kernel.routes_of(TUNNEL_INTERFACE), vec![cidr("0.0.0.0/0")]); +} + +/// A knob this kernel does not have is skipped, not fatal: IPv6 can be compiled +/// out, and a node with no IPv6 guests is still a working node. +#[tokio::test] +async fn a_kernel_without_ipv6_still_configures() { + let kernel = FakeKernel { + missing: vec![ + "net/ipv6/conf/all/forwarding".to_string(), + format!("net/ipv6/conf/{GUEST_BRIDGE}/proxy_ndp"), + ], + ..FakeKernel::new() + }; + apply(&kernel, &desired(), &key()).await.unwrap(); + assert_eq!( + kernel.sysctl_value("net/ipv4/ip_forward"), + Some("1".to_string()) + ); +} + +/// An address LNVPS sent that is not an address is reported against the value: +/// the node has to say which part of the document it could not use. +#[tokio::test] +async fn a_malformed_address_is_reported() { + let kernel = FakeKernel::new(); + let mut plane = desired(); + plane.tunnel.address4 = Some("not-an-address".to_string()); + let err = apply(&kernel, &plane, &key()).await.unwrap_err(); + assert!(format!("{err:#}").contains("not-an-address"), "{err:#}"); + + let mut plane = desired(); + plane.gateways = vec!["also-not".to_string()]; + let err = apply(&kernel, &plane, &key()).await.unwrap_err(); + assert!(format!("{err:#}").contains("also-not"), "{err:#}"); +} + +/// Observation reads the machine rather than remembering what was applied: the +/// point of observing is to catch the case where the two disagree. +#[tokio::test] +async fn observation_reports_what_the_machine_has() { + let kernel = FakeKernel::new(); + apply(&kernel, &desired(), &key()).await.unwrap(); + + // WireGuard comes up happily with a peer that never answers, so an interface + // that has never handshaken is configured, not working. + let state = observe(&kernel).await.unwrap(); + assert!(state.tunnel_up); + assert_eq!(state.tunnel_mtu, Some(1420)); + assert_eq!(state.last_handshake_secs, None); + assert!(state.bridge_up); + assert!(state.forwarding4 && state.forwarding6); + assert_eq!(state.routed_guests, 1); + assert!( + !state.healthy(), + "a tunnel that never handshook is not healthy" + ); + + // Once the route server has answered, it is. + kernel + .wireguard + .lock() + .unwrap() + .get_mut(TUNNEL_INTERFACE) + .unwrap() + .peers = vec![("peer".to_string(), Some(12))]; + let state = observe(&kernel).await.unwrap(); + assert_eq!(state.last_handshake_secs, Some(12)); + assert!(state.healthy()); +} + +/// A machine with nothing configured reports nothing configured rather than +/// failing: "not set up yet" is a state the health gate has to be able to read. +#[tokio::test] +async fn an_unconfigured_machine_observes_cleanly() { + let kernel = FakeKernel::new(); + let state = observe(&kernel).await.unwrap(); + assert_eq!(state, DataPlaneState::default()); + assert!(!state.healthy()); +} + +/// `/proc/sys` is read and written as files rather than through the `sysctl` +/// binary: one less program a node must have installed, and a write that either +/// happens or says why. +#[test] +fn kernel_knobs_are_files() { + let dir = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(dir.path().join("net/ipv4")).unwrap(); + std::fs::write(dir.path().join("net/ipv4/ip_forward"), "0\n").unwrap(); + + assert_eq!( + read_sysctl(dir.path(), "net/ipv4/ip_forward").unwrap(), + Some("0\n".to_string()) + ); + write_sysctl(dir.path(), "net/ipv4/ip_forward", "1").unwrap(); + assert_eq!( + read_sysctl(dir.path(), "net/ipv4/ip_forward").unwrap(), + Some("1".to_string()) + ); + + // A knob that is not there is a fact about the machine, not a failure. + assert_eq!(read_sysctl(dir.path(), "net/ipv6/absent").unwrap(), None); + assert!(write_sysctl(dir.path(), "net/ipv6/absent", "1").is_err()); +} diff --git a/lnvps_node/src/netns.rs b/lnvps_node/src/netns.rs new file mode 100644 index 00000000..6fc33e82 --- /dev/null +++ b/lnvps_node/src/netns.rs @@ -0,0 +1,218 @@ +//! The network namespace the node's data plane lives in. +//! +//! A marketplace node is somebody else's machine, and often not only an LNVPS +//! node. Configuring the data plane in the machine's own namespace means taking +//! their default route, turning on forwarding machine-wide, and putting guest +//! routes and proxy-ARP settings in the tables their own tooling manages. All +//! of that is rude, and some of it is dangerous. +//! +//! So the data plane gets a namespace of its own: +//! +//! - **Their default route stays theirs.** Ours is inside, where only guest +//! traffic sees it. +//! - **Guests cannot reach the operator's network at all.** Not because a +//! firewall rule says so — a rule can be mis-ordered, flushed by their +//! tooling, or forgotten on one address family — but because the namespace +//! holds no interface that leads there. +//! - **A tunnel that is down means no path at all.** Without this, a stray +//! route sends customer traffic out the operator's uplink sourced from LNVPS +//! addresses, which looks like spoofing to their upstream and can get *their* +//! connection null-routed. +//! - **Kernel knobs are scoped.** `ip_forward`, `proxy_arp` and `proxy_ndp` are +//! per-namespace, so LNVPS stops editing settings on their machine. +//! +//! The one thing that must stay outside is the tunnel's own UDP socket: a +//! WireGuard interface keeps its socket in the namespace it was *created* in, +//! so the tunnel interface is created in the machine's namespace and then moved +//! into this one. +//! The encrypted outer traffic still leaves through the operator's uplink, +//! while the inner interface — and everything routed over it — is isolated. + +use std::fs; +use std::os::fd::{AsFd, AsRawFd, OwnedFd}; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result}; +use nix::mount::{MntFlags, MsFlags, mount, umount2}; +use nix::sched::{CloneFlags, setns, unshare}; + +/// The namespace the data plane lives in. +pub const NAMESPACE: &str = "lnvps"; + +/// Where namespaces are pinned, matching iproute2 so `ip netns exec lnvps …` +/// works for an operator debugging their machine. +const NETNS_DIR: &str = "/run/netns"; + +/// The pinned path for a namespace. +pub fn path(root: &Path, name: &str) -> PathBuf { + root.join(name) +} + +/// Create the namespace if it does not exist, and return a handle to it. +/// +/// Pinned as a bind mount, exactly as `ip netns add` does it, so the namespace +/// outlives the daemon: a node restarting must not take its customers' network +/// with it, and an operator must be able to look inside with the tools they +/// already have. +pub fn ensure(root: &Path, name: &str) -> Result { + let pinned = path(root, name); + if pinned.exists() { + return Handle::open(&pinned); + } + + fs::create_dir_all(root) + .with_context(|| format!("Cannot create the namespace directory {}", root.display()))?; + fs::File::create(&pinned).with_context(|| { + format!( + "Cannot create the namespace mount point {}", + pinned.display() + ) + })?; + + // Done on a thread: unsharing changes the *calling thread's* namespace, and + // the daemon's other threads must keep the machine's own network — that is + // how it goes on reaching LNVPS while the tunnel is being built. + let pinned_for_thread = pinned.clone(); + std::thread::spawn(move || -> Result<()> { + unshare(CloneFlags::CLONE_NEWNET).context("Cannot create a network namespace")?; + // `/proc/thread-self`, not `/proc/self`: in a process with more than + // one thread, `/proc/self/ns/net` is the *process's* namespace, which + // is the one this thread just stopped being in. Pinning that would + // produce a "namespace" that is the machine's own network, and every + // isolation this module exists for would silently not happen. + mount( + Some(Path::new("/proc/thread-self/ns/net")), + &pinned_for_thread, + None::<&str>, + MsFlags::MS_BIND, + None::<&str>, + ) + .with_context(|| { + format!( + "Cannot pin the namespace at {}", + pinned_for_thread.display() + ) + })?; + Ok(()) + }) + .join() + .map_err(|_| anyhow::anyhow!("The thread creating the namespace panicked"))??; + + Handle::open(&pinned) +} + +/// Remove a pinned namespace. Used by tests and teardown, not in normal running. +pub fn remove(root: &Path, name: &str) -> Result<()> { + let pinned = path(root, name); + if !pinned.exists() { + return Ok(()); + } + // Unmounting drops the namespace once nothing else holds it; the file is + // then just a file. + let _ = umount2(&pinned, MntFlags::MNT_DETACH); + fs::remove_file(&pinned) + .with_context(|| format!("Cannot remove the namespace file {}", pinned.display())) +} + +/// The default pinned location, used outside tests. +pub fn ensure_default() -> Result { + ensure(Path::new(NETNS_DIR), NAMESPACE) +} + +/// An open network namespace. +#[derive(Debug)] +pub struct Handle { + fd: OwnedFd, + path: PathBuf, +} + +impl Handle { + /// Open an already-pinned namespace. + pub fn open(pinned: &Path) -> Result { + let file = fs::File::open(pinned) + .with_context(|| format!("Cannot open the namespace at {}", pinned.display()))?; + Ok(Self { + fd: OwnedFd::from(file), + path: pinned.to_path_buf(), + }) + } + + /// Where this namespace is pinned. + pub fn pinned_at(&self) -> &Path { + &self.path + } + + /// The descriptor, for the netlink call that moves an interface in here. + pub fn as_raw_fd(&self) -> std::os::fd::RawFd { + self.fd.as_fd().as_raw_fd() + } + + /// Run `f` with this namespace as the current one. + /// + /// On a thread of its own, because namespace membership is per-thread: the + /// daemon's other threads keep the machine's own network, which is what + /// lets it go on reaching LNVPS while the tunnel is down or being built. + /// The thread is scoped and joined here, so it ends inside the namespace + /// rather than being returned to a pool carrying it. + pub fn enter(&self, f: F) -> Result + where + F: FnOnce() -> Result + Send, + T: Send, + { + let target = self.fd.as_fd().as_raw_fd(); + std::thread::scope(|scope| { + scope + .spawn(move || -> Result { + setns_fd(target).context("Cannot enter the data plane namespace")?; + f() + }) + .join() + .map_err(|_| anyhow::anyhow!("A thread entering the namespace panicked"))? + }) + } +} + +/// `setns` on a raw fd, restricted to the network namespace. +fn setns_fd(fd: std::os::fd::RawFd) -> Result<()> { + // SAFETY: the fd is owned by the caller for the duration of the call. + let borrowed = unsafe { std::os::fd::BorrowedFd::borrow_raw(fd) }; + setns(borrowed, CloneFlags::CLONE_NEWNET)?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The whole point of the module: a thread inside the namespace must see a + /// different network from the machine's, and the namespace must outlive the + /// thread that made it so a daemon restart does not take the data plane + /// with it. + #[test] + #[ignore = "requires root; run with scripts/tunnel-e2e.sh"] + fn a_namespace_isolates_and_persists() { + let root = tempfile::tempdir().unwrap(); + let handle = ensure(root.path(), "lnvps-test").unwrap(); + + let outside = std::fs::read_link("/proc/thread-self/ns/net").unwrap(); + let inside = handle + .enter(|| Ok(std::fs::read_link("/proc/thread-self/ns/net")?)) + .unwrap(); + assert_ne!( + outside, inside, + "the thread stayed in the machine's network" + ); + + // The daemon's own threads keep the machine's network, which is what + // lets it go on reaching LNVPS while the tunnel is down. + assert_eq!( + std::fs::read_link("/proc/thread-self/ns/net").unwrap(), + outside + ); + + // Pinned, so it survives this process. + assert!(path(root.path(), "lnvps-test").exists()); + remove(root.path(), "lnvps-test").unwrap(); + assert!(!path(root.path(), "lnvps-test").exists()); + } +} diff --git a/lnvps_node/src/wgkey.rs b/lnvps_node/src/wgkey.rs new file mode 100644 index 00000000..05c8c120 --- /dev/null +++ b/lnvps_node/src/wgkey.rs @@ -0,0 +1,292 @@ +//! The node's WireGuard keypair. +//! +//! Generated here and kept here: LNVPS is told the public half and never sees +//! the private one. That is the whole reason a node presents a key rather than +//! being issued one — an operator's machine that LNVPS could impersonate would +//! make the tunnel's authentication decorative. +//! +//! The key is generated in-process rather than by shelling out to `wg genkey`, +//! so a node with a broken or missing `wg` fails when it tries to *configure* +//! the interface, with that error, instead of failing here with a confusing +//! one — and so the tests do not need a fake for key generation. +//! +//! These few lines duplicate `lnvps_api_common::wireguard` rather than +//! depending on it: that crate pulls in the database, axum and the payment +//! stack, none of which belong on somebody else's hardware. + +use std::fs; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, bail}; +use base64::Engine; +use base64::engine::general_purpose::STANDARD; +use rand::TryRngCore; +use x25519_dalek::{PublicKey, StaticSecret}; + +/// A node's WireGuard identity. +pub struct NodeKey { + secret: StaticSecret, + /// True when this run created the key, so the caller can say so once rather + /// than logging it on every start. + pub generated: bool, +} + +impl std::fmt::Debug for NodeKey { + /// Hand-written: the derived form would print the private key, and this + /// type travels through anyhow errors and log lines. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("NodeKey()") + } +} + +impl NodeKey { + /// The public half, base64, as WireGuard writes keys. + pub fn public_base64(&self) -> String { + STANDARD.encode(PublicKey::from(&self.secret).as_bytes()) + } + + /// The public half as raw bytes, which is how LNVPS stores it. + pub fn public_bytes(&self) -> [u8; 32] { + *PublicKey::from(&self.secret).as_bytes() + } + + /// The private half, base64. + /// + /// Only for handing to `wg` through a file; it is deliberately not + /// reachable from `Debug` or `Display`. + pub fn private_base64(&self) -> String { + STANDARD.encode(self.secret.to_bytes()) + } +} + +/// Where the key lives inside the state directory. +pub fn key_path(state_dir: &Path) -> PathBuf { + state_dir.join("tunnel.key") +} + +/// Load the node's key, generating one on first use. +/// +/// A regenerated key is not fatal: LNVPS re-pins a node that presents a new one +/// rather than refusing it, because a machine restored from backup that can +/// never be reached again is worse than a re-pin. The caller still warns, since +/// the tunnel stays down until the new key has been presented. +pub fn load_or_generate(state_dir: &Path) -> Result { + let path = key_path(state_dir); + if path.exists() { + crate::credential::check_permissions(&path)?; + let contents = fs::read_to_string(&path) + .with_context(|| format!("Cannot read tunnel key {}", path.display()))?; + return parse(&contents, &path); + } + + fs::create_dir_all(state_dir) + .with_context(|| format!("Cannot create state directory {}", state_dir.display()))?; + let mut bytes = [0u8; 32]; + rand::rngs::OsRng + .try_fill_bytes(&mut bytes) + .map_err(|e| anyhow::anyhow!("No system randomness available for a tunnel key: {e}"))?; + let secret = StaticSecret::from(bytes); + write_secret(&path, &STANDARD.encode(secret.to_bytes()))?; + Ok(NodeKey { + secret, + generated: true, + }) +} + +/// Parse a stored key. +pub fn parse(contents: &str, path: &Path) -> Result { + let raw = STANDARD + .decode(contents.trim()) + .with_context(|| format!("Tunnel key {} is not base64", path.display()))?; + let bytes: [u8; 32] = raw.as_slice().try_into().map_err(|_| { + anyhow::anyhow!( + "Tunnel key {} is {} bytes; a WireGuard key is 32", + path.display(), + raw.len() + ) + })?; + Ok(NodeKey { + secret: StaticSecret::from(bytes), + generated: false, + }) +} + +/// Write `contents` to `path` readable only by its owner. +/// +/// The mode is set **before** the key is written, not after: a key that is +/// world-readable for even the moment between the two is a key that a process +/// watching the directory has already read. +#[cfg(unix)] +fn write_secret(path: &Path, contents: &str) -> Result<()> { + use std::io::Write; + use std::os::unix::fs::OpenOptionsExt; + + let mut file = fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .open(path) + .with_context(|| format!("Cannot write tunnel key {}", path.display()))?; + file.write_all(contents.as_bytes())?; + Ok(()) +} + +/// Non-Unix hosts have no mode bits. Nodes are Linux-only; this exists so the +/// crate still builds on a developer's other machine. +#[cfg(not(unix))] +fn write_secret(path: &Path, contents: &str) -> Result<()> { + fs::write(path, contents).with_context(|| format!("Cannot write tunnel key {}", path.display())) +} + +/// Write the private key to a file `wg` can read, inside the state directory. +/// +/// `wg set` takes the key as a **path**, never as an argument, because an +/// argument is visible in `ps` to every user on the machine — and a marketplace +/// node usually has more than one login. +pub fn write_private_key_file(state_dir: &Path, key: &NodeKey) -> Result { + let path = state_dir.join("tunnel.key.wg"); + write_secret(&path, &key.private_base64())?; + Ok(path) +} + +/// Reject a server key that is not a WireGuard key, before it reaches `wg`. +pub fn parse_public_key(value: &str) -> Result { + // LNVPS sends hex; `wg` speaks base64. Converting here keeps the wire + // format consistent with the rest of the node API, where keys are hex. + let raw = hex::decode(value.trim()) + .with_context(|| format!("Server public key {value} is not hex"))?; + if raw.len() != 32 { + bail!( + "Server public key is {} bytes; a WireGuard key is 32", + raw.len() + ); + } + Ok(STANDARD.encode(raw)) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A key generated here must be the one that comes back, and the public + /// half must be derived from it rather than stored beside it — a stored + /// copy is free to disagree with the key it claims to describe. + #[test] + fn a_key_survives_a_restart() { + let dir = tempfile::tempdir().unwrap(); + let first = load_or_generate(dir.path()).unwrap(); + assert!(first.generated); + + let second = load_or_generate(dir.path()).unwrap(); + assert!(!second.generated, "a second start generated a new key"); + assert_eq!(first.public_base64(), second.public_base64()); + assert_eq!(first.private_base64(), second.private_base64()); + assert_eq!(first.public_bytes(), second.public_bytes()); + } + + /// The key is written owner-only from the moment it exists: a marketplace + /// node usually has more than one login on it. + #[cfg(unix)] + #[test] + fn a_generated_key_is_not_readable_by_other_users() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + load_or_generate(dir.path()).unwrap(); + let mode = fs::metadata(key_path(dir.path())) + .unwrap() + .permissions() + .mode(); + assert_eq!(mode & 0o077, 0, "mode {:04o}", mode & 0o7777); + } + + /// A key file that is readable by everyone is refused rather than used: the + /// tunnel it authenticates is the node's whole security boundary. + #[cfg(unix)] + #[test] + fn a_world_readable_key_is_refused() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + load_or_generate(dir.path()).unwrap(); + fs::set_permissions(key_path(dir.path()), fs::Permissions::from_mode(0o644)).unwrap(); + assert!(load_or_generate(dir.path()).is_err()); + } + + /// The key must not reach a log line or an anyhow error, both of which + /// this type travels through. + #[test] + fn the_key_is_never_printed() { + let dir = tempfile::tempdir().unwrap(); + let key = load_or_generate(dir.path()).unwrap(); + let printed = format!("{key:?}"); + assert!(!printed.contains(&key.private_base64()), "{printed}"); + assert!(printed.contains("redacted"), "{printed}"); + } + + /// A state directory that cannot hold a key, and a key file that cannot be + /// read, are reported against the path rather than as a bare io error — the + /// operator has to know which file to fix. + #[test] + fn an_unusable_state_directory_is_reported_with_its_path() { + let dir = tempfile::tempdir().unwrap(); + + // A file where the state directory should be. + let as_file = dir.path().join("state"); + fs::write(&as_file, "").unwrap(); + let err = load_or_generate(&as_file).unwrap_err(); + assert!(format!("{err:#}").contains("state"), "{err:#}"); + + // A directory where the key file should be: it exists, so it is read, + // and reading it fails. + let state = dir.path().join("other"); + fs::create_dir_all(key_path(&state)).unwrap(); + let err = load_or_generate(&state).unwrap_err(); + assert!(format!("{err:#}").contains("tunnel.key"), "{err:#}"); + } + + /// Anything that is not 32 bytes of base64 is not a WireGuard key, and + /// saying so here beats a handshake that never completes. + #[test] + fn a_malformed_key_is_refused() { + let path = Path::new("/tmp/tunnel.key"); + assert!(parse("not base64!", path).is_err()); + assert!(parse(&STANDARD.encode([1u8; 16]), path).is_err()); + assert!(parse(&STANDARD.encode([1u8; 32]), path).is_ok()); + } + + /// The server's key arrives as hex and reaches `wg` as base64; a value that + /// is neither is refused before it is written into a command. + #[test] + fn a_server_key_is_converted_and_checked() { + assert_eq!( + parse_public_key(&hex::encode([0xab; 32])).unwrap(), + STANDARD.encode([0xab; 32]) + ); + assert!(parse_public_key("nonsense").is_err()); + assert!(parse_public_key(&hex::encode([0xab; 16])).is_err()); + } + + /// `wg set` takes a path, never an argument: an argument is visible in `ps` + /// to every user on the machine. + #[cfg(unix)] + #[test] + fn the_key_file_handed_to_wg_is_owner_only() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + let key = load_or_generate(dir.path()).unwrap(); + let path = write_private_key_file(dir.path(), &key).unwrap(); + assert_eq!(fs::read_to_string(&path).unwrap(), key.private_base64()); + let mode = fs::metadata(&path).unwrap().permissions().mode(); + assert_eq!(mode & 0o077, 0, "mode {:04o}", mode & 0o7777); + + // A key file that cannot be written is named, not swallowed: `wg` would + // otherwise fail later with a path the operator never chose. + let blocked = tempfile::tempdir().unwrap(); + fs::create_dir_all(blocked.path().join("tunnel.key.wg")).unwrap(); + let err = write_private_key_file(blocked.path(), &key).unwrap_err(); + assert!(format!("{err:#}").contains("tunnel.key.wg"), "{err:#}"); + } +} diff --git a/lnvps_node/tests/control_https.rs b/lnvps_node/tests/control_https.rs index d15584e5..289e5422 100644 --- a/lnvps_node/tests/control_https.rs +++ b/lnvps_node/tests/control_https.rs @@ -32,7 +32,13 @@ async fn start_node(keys: &Keys, state_dir: &std::path::Path) -> (SocketAddr, Ve let node_tls = tls::load_or_generate(state_dir, Some(addr.ip())).unwrap(); let (cert, fingerprint) = (node_tls.cert_pem.clone(), node_tls.fingerprint.clone()); - let state = Arc::new(ControlState::new(keys.public_key(), addr)); + // The data plane this reports on is the machine's; these tests are about + // TLS and authentication, so an unconfigured machine is the honest answer. + let state = Arc::new(ControlState::new( + keys.public_key(), + addr, + Arc::new(lnvps_node::net::UnavailableKernel), + )); tokio::spawn(async move { serve(state, addr, node_tls).await }); // Wait for the listener rather than sleeping a fixed time. diff --git a/scripts/tunnel-e2e.sh b/scripts/tunnel-e2e.sh new file mode 100755 index 00000000..fc7ff6d6 --- /dev/null +++ b/scripts/tunnel-e2e.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# tunnel-e2e.sh — Run the marketplace tunnel harness: both ends, real kernel. +# +# The harness (lnvps_e2e/tests/tunnel_netns.rs) builds a route server and a node +# out of network namespaces, configures each end with the real production code +# paths, and pings across the tunnel — including to a guest sitting behind the +# node, which is the path a customer's traffic takes. +# +# It needs root (namespaces, veth, WireGuard), so the tests are #[ignore]d and +# only run from here. +# +# Usage: +# ./scripts/tunnel-e2e.sh [--filter NAME] + +set -euo pipefail + +FILTER="" +while [[ $# -gt 0 ]]; do + case "$1" in + --filter) FILTER="$2"; shift 2 ;; + --) shift; break ;; + *) echo "Unknown option: $1" >&2; exit 1 ;; + esac +done + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$(cd "$SCRIPT_DIR/.." && pwd)" + +# Built as the invoking user so the build cache and toolchain resolve against +# the normal $HOME; only the run needs to be privileged. +echo "=== Building the tunnel harness ===" +cargo test -p lnvps_e2e --test tunnel_netns --no-run + +CMD=(cargo test -p lnvps_e2e --test tunnel_netns -- --ignored --test-threads=1) +[[ -n "$FILTER" ]] && CMD+=("$FILTER") +[[ $# -gt 0 ]] && CMD+=("$@") + +echo "=== Running the tunnel harness as root ===" +if [[ "$(id -u)" -eq 0 ]]; then + "${CMD[@]}" +else + sudo -E "${CMD[@]}" +fi diff --git a/work/marketplace.md b/work/marketplace.md index 83ea27b3..98464e6f 100644 --- a/work/marketplace.md +++ b/work/marketplace.md @@ -99,7 +99,7 @@ Node is a **client**, never a server: ### Data plane (all traffic over WireGuard) ``` -guest VM ─ tap ─ br-lnvps (no operator uplink) ─ wg0 ─┐ +guest VM ─ tap ─ br-lnvps (no operator uplink) ─ wgln0 ─┐ │ WG (UDP) operator NAT / any ISP │ ▼ @@ -744,7 +744,7 @@ What the build settled beyond the plan: constraint, which is exactly the pool-less case. The mock mirrors it. - **A node takes one address, not a link** (revised during 4b; 4a shipped /31s and /127s). WireGuard is layer 3 and point-to-point, with no ARP and no on-link requirement, so the node - needs no gateway of its own — `ip route add default dev wg0` is enough. A /31 therefore spent + needs no gateway of its own — `ip route add default dev wgln0` is enough. A /31 therefore spent two addresses describing something that needs one, and worse, forced the route server to hold one address per node on a single interface: a /16 pool with a thousand nodes meant a thousand addresses on `wgln`, re-parsed out of `ip addr show` on every reconcile. The route server @@ -831,12 +831,126 @@ for want of a real box, as before. from a router is drift to report, not an allocation to forget. - Route the guest prefixes at the route server towards the peer. -#### 4c — Node data plane + health gate (L) ⬅ NEXT -- Node side: `wg0` + `br-lnvps`, default route into the tunnel, anti-spoof and anti-LAN-access - rules, MTU/MSS clamp. -- Health gate: the host is only enabled after an end-to-end reachability probe from the route - server through the tunnel to a test guest IP. Failure leaves the node approved but unusable, - which is the safe direction. +#### 4c — Node data plane + health gate (XL — split into 4c1/4c2/4c3) + +Sized as one increment it is XL. Three decisions taken before starting made it so, each +deliberately: + +- **The daemon applies the configuration itself**, with `ip` and `wg`, rather than writing + `wg-quick` files for something else to read. A marketplace node runs on hardware LNVPS does + not own, so a data plane that depends on the operator having wired it up correctly is one + whose mistakes surface as a customer's VM having no network. The daemon re-converges instead. +- **Both `nft` and `iptables` are supported**, detected at runtime, because a node is somebody + else's machine and refusing the ones that run iptables would refuse real hardware. It costs a + second dialect, which is why the firewall is its own increment. +- **The health gate spawns a real guest and pings it** rather than asking the node how it + thinks it is doing. The node self-reporting "bridge up, forwarding on" cannot catch a bridge + with no path to the tunnel, which is exactly the mistake worth catching before a customer + finds it. + +The node also has no outbound API client yet (that was 2c, still pending), so 4c1 builds the +one call it needs rather than waiting. + +#### 4c1 — Node data plane (L) ✅ DONE + +What the build settled beyond the plan: +- **The data plane is applied before the listener binds.** The control API binds an address of + the tunnel interface, and on a fresh machine that interface does not exist until the daemon + has fetched and applied its document — so the startup order is apply, then check, then serve. + A failure to fetch is a warning rather than a fatal error: a node whose tunnel is already up + from a previous run must keep serving through an LNVPS outage, or an API blip takes every node + on the platform dark at once. +- **The gateway a guest uses belongs to its range, not to the node.** The guest is configured + with the range's gateway and believes it is on-link, so the node holds that address on the + bridge as a **host** address and turns on proxy ARP/NDP. Holding the whole range instead would + make the node believe every other node's guests were local, and their traffic would disappear + into the bridge instead of going up the tunnel. +- **The bridge takes the tunnel's MTU.** A guest sending 1500 bytes into a 1420-byte tunnel gets + a connection that opens and then hangs on the first large transfer — the worst failure shape + there is, because everything looks fine until it does not. +- **A peer that is not the route server is removed from the tunnel interface**, most likely a stale key left by + a re-key, which would otherwise still be able to send traffic the node treats as LNVPS's. +- **Routes for departed guests are swept**, since a released address goes straight back in the + pool and may already be somebody else's; the bridge's own gateway addresses are excluded from + that sweep so tidying up after a guest cannot take the bridge's addressing with it. +- **Observation reads the machine, never a cache.** `/api/v1/status` runs the queries on + demand: a cached answer reports that the tunnel was up once, which is exactly what the health + gate must not accept. A tunnel that has never handshaken is reported as configured but not + working, because WireGuard comes up perfectly happily with a peer that never answers. +- **The node generates its key in-process** rather than shelling out to `wg genkey`, so a + missing `wg` fails when the interface is configured, with that error. The private key reaches + `wg` as a **path**, never an argument: arguments are visible in `ps` to every user on the + machine, and a marketplace node usually has more than one login. +- **`lnvps-node dataplane observe` deliberately needs no credential**, because "what does this + machine actually have?" is the question asked when something is already broken. + +Reworked during review, before merge: +- **Netlink, not `ip`.** The daemon speaks to the kernel directly (`rtnetlink` for links, + addresses and routes; the WireGuard netlink interface for the tunnel; `/proc/sys` for the + forwarding knobs). `ip` is a program that formats netlink messages and formats the answers + back into text for us to parse; going direct removes a dependency on iproute2's presence and + version, the output parsing that changes between releases, and gives kernel error codes + instead of a line of English on stderr. +- **The data plane lives in its own network namespace.** The first cut configured the + *machine's* network: it took the operator's default route and turned on forwarding + machine-wide, on hardware that is often not only an LNVPS node. Now `wg0` and `br-lnvps` live + in an `lnvps` namespace, so their default route stays theirs, the forwarding and proxy-ARP + knobs are ours alone, and guests cannot reach the operator's network — not because a rule + forbids it, but because no interface leads there. A tunnel that is down means no path at all, + rather than customer traffic leaking out the operator's uplink sourced from LNVPS addresses, + which looks like spoofing to their upstream. `wg0` is created in the machine's namespace and + *moved*, because a WireGuard interface keeps its UDP socket where it was created — that is + what lets the encrypted outer traffic still use the operator's uplink. +- **The node's tunnel is `wgln0`, not `wg0`.** It is created in the *machine's* namespace + before being moved into the data plane's, so the name has to be one an operator is not + already using — a VPN, a mesh, anything called `wg0` would either fail the creation or, worse, + be adopted and moved out from under them. The `wgln` prefix is the same one the route server + uses, so a managed interface is recognisable as LNVPS's wherever it appears. The harness + proves it by putting an operator's own `wg0` on the machine first and checking it survives. +- **The bridge name is no longer sent.** Both sides hold it as a constant, because the daemon + needs the name before it has ever spoken to LNVPS (`dataplane observe` takes no credential), + and a document that could name a different one would leave the node holding two answers. The + harness asserts the two constants agree. + +Testing note: the orchestration is tested against a fake kernel behind a `NetOps` trait — what +is worth asserting is what the node *decides*. Whether those decisions work is proven by +`lnvps_e2e/tests/tunnel_netns.rs`, which builds both ends in network namespaces and pings +across the tunnel, including to a guest behind the node. It found four bugs nothing else did: +a namespace pinned from `/proc/self/ns/net` (the *process's* namespace in a threaded program, +so every "isolated" interface silently landed in the operator's network), WireGuard netlink +calls made outside the namespace the interface had been moved into, local-table routes being +mistaken for strays and deleted — and, in already-merged 4b code, **the route server never +routing the pool's own block**, so a route server holding `10.66.0.1/16` answered "network is +unreachable" for every node in the pool. + +#### 4c1 — original scope +- `GET /api/v1/node/dataplane` (node token): the desired state in one document — the tunnel + (key, addresses, endpoint, MTU, keepalive), the bridge, and the guest addresses assigned to + this node. One call rather than three: the node applies these together or not at all, and a + document that can be half-fetched is a data plane that can be half-applied. +- Node keypair: generated on first use into the state directory, `0600`, public half presented + to `POST /api/v1/node/tunnel`. The private half never leaves the machine. +- `net.rs`: `wg0`, `br-lnvps`, the default route into the tunnel, MTU, and IP forwarding — + idempotent, applied on startup and on every refresh, through a command runner that tests can + record. These commands run as root on somebody else's machine; the exact command is the thing + worth asserting. +- `lnvps-node dataplane show|apply` so an operator can see and re-drive it without the daemon. +- `/api/v1/status` reports the observed data plane, which 4c3's gate reads as its first check. + +#### 4c2 — Anti-spoof + anti-LAN firewall (M/L) ⬅ NEXT +- One ruleset the daemon owns wholesale, in `nft` where available and `iptables` where not, + with the backend detected once and reported in status. +- Guests may not reach the operator's own LAN or the node's management addresses; may not + source traffic as an address LNVPS did not assign them; MSS clamped to the tunnel's MTU. +- The guest address set comes from 4c1's document, so the boundary is LNVPS's list, not + something the node infers. + +#### 4c3 — Health gate (M/L) +- A probe guest is provisioned through the ordinary path onto the new node, given a real + address, and pinged from the route server through the tunnel. Only then is the host enabled. +- Failure leaves the node approved but unusable, with the failing step named. That is the safe + direction: a node that never carries a customer is a support conversation, a node that + carries one badly is an outage. ### Increment 5 — Confidential computing: attestation + encrypted disks (L) - Verify SEV-SNP attestation reports (`sev` crate) / TDX quotes (`dcap-qvl` crate) against