feat(marketplace): realise a node's tunnel on the route server - #365
Conversation
An allocated tunnel was paperwork: addresses and a key written down, and nothing configured anywhere. The node's peer is now pushed to the route server, so the tunnel carries traffic. **The peer's AllowedIPs is a security boundary, not a routing hint.** It is the node's own inner addresses plus exactly the guest addresses LNVPS assigned to it. WireGuard drops an inbound packet whose source is not on that list, so a node cannot source traffic as another node's customer, and `wg set peer allowed-ips` *replaces* the list — a released address stops being accepted on the next reconcile rather than lingering. **AllowedIPs is not a route.** It decides which peer a packet already headed down the tunnel belongs to; it does not put the packet there. Without an explicit route the guest's return traffic reaches the route server and is dropped as unroutable, so guest prefixes are routed down the interface and the route server takes an address on each point-to-point link. Peers are pushed one at a time rather than through `update_tunnel`, which recreates the interface on the Linux backend and takes every peer with it: one node getting a guest address must not cut every other node on the route server. `TunnelRouter` therefore grew four methods — `set_tunnel_peer`, `remove_tunnel_peer`, `sync_tunnel_addresses`, `sync_tunnel_routes` — which a Mikrotik refuses outright rather than accepting a pool and configuring nothing. Drift is reported, not just repaired. `missing`, `changed` and `unclaimed` are kept apart because they mean different things: a peer that is gone was configured and vanished, a changed one is carrying the wrong anti-spoof list, and an unclaimed one is a key on an LNVPS interface that no allocation accounts for. Unclaimed peers are removed — `wgln*` is ours outright, so a key there is either a revoked node or somebody else's. Two things the implementation had to get right rather than assume: - **A reconcile must not touch what the kernel owns.** The IPv6 link-local address and the /31 link route are the kernel's; deleting them to tidy a list would break the interface, on every single poll. Both are excluded, and both address families are queried separately because `ip route show` is IPv4 only — a v6 guest prefix would otherwise look absent forever. - **Allowed IPs are compared as a set.** `wg` reports them in its own order, and treating that as a difference would rewrite a working peer's boundary on every poll. Correctness does not depend on a job firing: the reconcile runs on the existing router poll, so a peer wiped by a reboot or an address assigned since the last push is fixed without an admin doing anything. `SyncNodeTunnel` exists only so a node asking for its tunnel does not wait behind every other node on the box. The backing host stays disabled. A configured peer is still not a proven path; the health gate that enables it comes with the node-side data plane. `LinuxSshRouter` gained a `#[cfg(test)]` command hook: these methods run commands as root on somebody else's route server, so what is worth asserting is the exact command issued, which needs the transport replaced rather than mocked around.
| /// the unimplemented half of the capability says so instead. | ||
| #[tokio::test] | ||
| async fn peer_level_operations_are_refused_rather_than_ignored() { | ||
| let r = MikrotikRouter::new("http://10.0.0.1", "admin", "pw"); |
4a gave every node a /31 and put the route server's half of each link on the pool's interface. That is one address on `wgln<id>` for every node on the route server — thousands of them on a /16 pool, re-parsed out of `ip addr show` on every reconcile — to describe something WireGuard does not need described. WireGuard is layer 3 and point-to-point: no ARP, no neighbour resolution, no on-link requirement. A node needs no gateway address of its own at all; `ip route add default dev wg0` is enough. So the /31 was spending two addresses to say what one address says, and charging the route server per node for the privilege. A node now holds a /32 (or /128), and the route server holds **one** address for the whole pool, carrying the block's own prefix so every node in it is on-link. The interface goes from O(nodes) addresses to exactly one per family, and a pool's usable capacity roughly doubles. Reserved addresses follow from the route server holding the block on-link: its network address, the route server's address immediately after it, and — on IPv4 — its broadcast address are not the pool's to hand out, because they are addresses the route server itself will not forward to. A /24 therefore places 253 nodes, and block validation now asks what a block can actually place rather than comparing prefixes: a /31 looks like it holds two nodes and holds none. `gateway4`/`gateway6` in the node-facing response are unchanged in shape but now derive from the pool's block instead of the node's link, so every node on a pool is told the same gateway. Nothing about peers, AllowedIPs or the anti-spoof boundary changes. Done now rather than later because no node holds a /31 in production yet; afterwards this would be a re-addressing exercise on live machines.
|
Second commit: a node now takes one address, not a point-to-point link. 4a gave each node a
Doing it now because no node holds a Docs and the changelog note the response change ( |
Increment 4b of the marketplace work (
work/marketplace.md). 4a allocated a tunnel — addresses and a key written down, nothing configured anywhere. This makes the tunnel carry traffic.What it does
TunnelRouter, on allocation and on the routine router poll.AllowedIPs= the node's inner addresses + exactly its assigned guest addresses. That is the anti-spoof boundary, not a routing hint: WireGuard drops an inbound packet whose source is not listed, so a node cannot source traffic as another node's customer.wg set peer allowed-ipsreplaces the list, so a released guest address stops being accepted on the next reconcile.AllowedIPsdecides which peer a packet already headed for the tunnel belongs to — it does not put the packet there, so without a route the guest's return traffic is dropped as unroutable.missing/changed/unclaimed. They mean different things: a peer that is gone was configured and vanished, a changed one is carrying the wrong anti-spoof list, and an unclaimed one is a key on an LNVPS interface that no allocation accounts for. Unclaimed peers are removed —wgln*is ours outright.Why peers are pushed one at a time
update_tunnelrecreates the interface on the Linux backend and takes every peer with it, so one node getting a guest address would cut every other node on the route server.TunnelRoutergainedset_tunnel_peer/remove_tunnel_peer/sync_tunnel_addresses/sync_tunnel_routes(wg set peer— additive and idempotent). A Mikrotik refuses them rather than accepting a pool and silently configuring nothing.Worth a reviewer's eye
ip route showis IPv4 only; a v6 guest prefix would otherwise look absent forever and be re-added on every sync.wgreports them in its own order. Treating that as a difference would rewrite a working peer's security boundary every poll.SyncNodeTunnelexists only for promptness.SyncTunnelPool's job, and doing it here would hide the fact that it never ran.Tests
LinuxSshRoutergained a#[cfg(test)]command hook — these run as root on somebody else's route server, so what is worth asserting is the exact command issued, which needs the transport replaced rather than mocked around. New tests cover the peer/address/route commands (including the kernel-owned entries being left alone and both families being queried), the plan builder (anti-spoof list, released addresses, deleted VMs, non-node tunnels), and the worker paths (realisation, drift repair in all three directions, refusing a missing interface, single-peer push and withdrawal, pool-less tunnel refused).Whole workspace suite green; 100% function coverage on the new code.