Skip to content

feat(marketplace): realise a node's tunnel on the route server - #365

Merged
v0l merged 2 commits into
masterfrom
feat/marketplace-tunnel-realise
Aug 6, 2026
Merged

feat(marketplace): realise a node's tunnel on the route server#365
v0l merged 2 commits into
masterfrom
feat/marketplace-tunnel-realise

Conversation

@v0l

@v0l v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Increment 4b of the marketplace work (work/marketplace.md). 4a allocated a tunnel — addresses and a key written down, nothing configured anywhere. This makes the tunnel carry traffic.

What it does

  • Pushes the node's peer to the route server through the existing TunnelRouter, on allocation and on the routine router poll.
  • AllowedIPs = the node's inner addresses + exactly its assigned guest addresses. That is the anti-spoof boundary, not a routing hint: WireGuard drops an inbound packet whose source is not listed, so a node cannot source traffic as another node's customer. wg set peer allowed-ips replaces the list, so a released guest address stops being accepted on the next reconcile.
  • Routes the guest prefixes down the interface and gives the route server an address on each point-to-point link. AllowedIPs decides which peer a packet already headed for the tunnel belongs to — it does not put the packet there, so without a route the guest's return traffic is dropped as unroutable.
  • Reports drift as missing / changed / unclaimed. They mean different things: a peer that is gone was configured and vanished, a changed one is carrying the wrong anti-spoof list, and an unclaimed one is a key on an LNVPS interface that no allocation accounts for. Unclaimed peers are removedwgln* is ours outright.

Why peers are pushed one at a time

update_tunnel recreates the interface on the Linux backend and takes every peer with it, so one node getting a guest address would cut every other node on the route server. TunnelRouter gained set_tunnel_peer / remove_tunnel_peer / sync_tunnel_addresses / sync_tunnel_routes (wg set peer — additive and idempotent). A Mikrotik refuses them rather than accepting a pool and silently configuring nothing.

Worth a reviewer's eye

  • The reconcile never deletes what the kernel owns — the v6 link-local address and the /31 link route — or it would fight the kernel on every poll. Both address families are queried separately because ip route show is IPv4 only; a v6 guest prefix would otherwise look absent forever and be re-added on every sync.
  • Allowed IPs are compared as a set, since wg reports them in its own order. Treating that as a difference would rewrite a working peer's security boundary every poll.
  • Correctness does not depend on a job firing. The reconcile rides the existing router poll, so a peer wiped by a reboot or an address assigned since the last push is corrected without an admin acting. SyncNodeTunnel exists only for promptness.
  • Reconcile refuses to create the interface — that is SyncTunnelPool's job, and doing it here would hide the fact that it never ran.
  • The backing host stays disabled: a configured peer is not a proven path. The health gate arrives with the node-side data plane (4c).

Tests

LinuxSshRouter gained a #[cfg(test)] command hook — these run as root on somebody else's route server, so what is worth asserting is the exact command issued, which needs the transport replaced rather than mocked around. New tests cover the peer/address/route commands (including the kernel-owned entries being left alone and both families being queried), the plan builder (anti-spoof list, released addresses, deleted VMs, non-node tunnels), and the worker paths (realisation, drift repair in all three directions, refusing a missing interface, single-peer push and withdrawal, pool-less tunnel refused).

Whole workspace suite green; 100% function coverage on the new code.

An allocated tunnel was paperwork: addresses and a key written down, and
nothing configured anywhere. The node's peer is now pushed to the route server,
so the tunnel carries traffic.

**The peer's AllowedIPs is a security boundary, not a routing hint.** It is the
node's own inner addresses plus exactly the guest addresses LNVPS assigned to
it. WireGuard drops an inbound packet whose source is not on that list, so a
node cannot source traffic as another node's customer, and `wg set peer
allowed-ips` *replaces* the list — a released address stops being accepted on
the next reconcile rather than lingering.

**AllowedIPs is not a route.** It decides which peer a packet already headed
down the tunnel belongs to; it does not put the packet there. Without an
explicit route the guest's return traffic reaches the route server and is
dropped as unroutable, so guest prefixes are routed down the interface and the
route server takes an address on each point-to-point link.

Peers are pushed one at a time rather than through `update_tunnel`, which
recreates the interface on the Linux backend and takes every peer with it: one
node getting a guest address must not cut every other node on the route server.
`TunnelRouter` therefore grew four methods — `set_tunnel_peer`,
`remove_tunnel_peer`, `sync_tunnel_addresses`, `sync_tunnel_routes` — which a
Mikrotik refuses outright rather than accepting a pool and configuring nothing.

Drift is reported, not just repaired. `missing`, `changed` and `unclaimed` are
kept apart because they mean different things: a peer that is gone was
configured and vanished, a changed one is carrying the wrong anti-spoof list,
and an unclaimed one is a key on an LNVPS interface that no allocation accounts
for. Unclaimed peers are removed — `wgln*` is ours outright, so a key there is
either a revoked node or somebody else's.

Two things the implementation had to get right rather than assume:

- **A reconcile must not touch what the kernel owns.** The IPv6 link-local
  address and the /31 link route are the kernel's; deleting them to tidy a list
  would break the interface, on every single poll. Both are excluded, and both
  address families are queried separately because `ip route show` is IPv4 only
  — a v6 guest prefix would otherwise look absent forever.
- **Allowed IPs are compared as a set.** `wg` reports them in its own order, and
  treating that as a difference would rewrite a working peer's boundary on every
  poll.

Correctness does not depend on a job firing: the reconcile runs on the existing
router poll, so a peer wiped by a reboot or an address assigned since the last
push is fixed without an admin doing anything. `SyncNodeTunnel` exists only so a
node asking for its tunnel does not wait behind every other node on the box.

The backing host stays disabled. A configured peer is still not a proven path;
the health gate that enables it comes with the node-side data plane.

`LinuxSshRouter` gained a `#[cfg(test)]` command hook: these methods run
commands as root on somebody else's route server, so what is worth asserting is
the exact command issued, which needs the transport replaced rather than mocked
around.
/// the unimplemented half of the capability says so instead.
#[tokio::test]
async fn peer_level_operations_are_refused_rather_than_ignored() {
let r = MikrotikRouter::new("http://10.0.0.1", "admin", "pw");
4a gave every node a /31 and put the route server's half of each link on the
pool's interface. That is one address on `wgln<id>` for every node on the route
server — thousands of them on a /16 pool, re-parsed out of `ip addr show` on
every reconcile — to describe something WireGuard does not need described.

WireGuard is layer 3 and point-to-point: no ARP, no neighbour resolution, no
on-link requirement. A node needs no gateway address of its own at all; `ip
route add default dev wg0` is enough. So the /31 was spending two addresses to
say what one address says, and charging the route server per node for the
privilege.

A node now holds a /32 (or /128), and the route server holds **one** address
for the whole pool, carrying the block's own prefix so every node in it is
on-link. The interface goes from O(nodes) addresses to exactly one per family,
and a pool's usable capacity roughly doubles.

Reserved addresses follow from the route server holding the block on-link: its
network address, the route server's address immediately after it, and — on IPv4
— its broadcast address are not the pool's to hand out, because they are
addresses the route server itself will not forward to. A /24 therefore places
253 nodes, and block validation now asks what a block can actually place rather
than comparing prefixes: a /31 looks like it holds two nodes and holds none.

`gateway4`/`gateway6` in the node-facing response are unchanged in shape but now
derive from the pool's block instead of the node's link, so every node on a pool
is told the same gateway. Nothing about peers, AllowedIPs or the anti-spoof
boundary changes.

Done now rather than later because no node holds a /31 in production yet;
afterwards this would be a re-addressing exercise on live machines.
@v0l

v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Second commit: a node now takes one address, not a point-to-point link.

4a gave each node a /31 and put the route server's half of every link on the pool's interface — one address on wgln<id> per node, thousands of them on a /16 pool, re-parsed out of ip addr show on every reconcile. WireGuard is layer 3 and point-to-point: no ARP, no neighbour resolution, no on-link requirement, and the node needs no gateway of its own (ip route add default dev wg0 suffices). The /31 was spending two addresses to say what one says, and charging the route server per node for it.

  • Node holds a /32 / /128; the route server holds one address per pool, carrying the block's own prefix so every node in it is on-link.
  • Interface addressing goes from O(nodes) to exactly one per family, and usable capacity roughly doubles.
  • Reserved addresses follow from holding the block on-link: its network address, the route server's address after it, and (IPv4) its broadcast address. A /24 places 253 nodes where it previously reported 128 links.
  • Block validation now asks what a block can actually place instead of comparing prefixes — a /31 looks like it holds two nodes and holds none.
  • gateway4/gateway6 keep their shape but derive from the pool's block, so every node on a pool is told the same gateway. Peers, AllowedIPs and the anti-spoof boundary are untouched.

Doing it now because no node holds a /31 in production yet; later it would be a re-addressing exercise on live machines.

Docs and the changelog note the response change (address4/address6 are now host prefixes, gateway* is pool-wide, capacity is counted in placeable addresses). Whole workspace suite green.

@v0l
v0l merged commit 3ef017f into master Aug 6, 2026
9 of 10 checks passed
@v0l
v0l deleted the feat/marketplace-tunnel-realise branch August 6, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants