feat(networking): Windows L2Bridge egress with router-safe VFP ACLs (opt-in) - #144
Conversation
On-metal validation —
|
| ACL set | Grafana :3000 |
Incus :8443 |
Proxmox .1/.2:8006 |
Router 192.168.1.1:80/443 |
ephemerd host :135 |
1.1.1.1:443 |
8.8.8.8:53 |
DNS by name | public HTTPS |
|---|---|---|---|---|---|---|---|---|---|
| none (baseline control) | OPEN | OPEN | OPEN | OPEN | OPEN | OPEN | OPEN | OK | HTTP 200 |
| allow-any only | OPEN | OPEN | OPEN | OPEN | OPEN | OPEN | OPEN | OK | HTTP 200 |
| ladder as generated | blocked | blocked | blocked | blocked | blocked | blocked | blocked | FAIL | FAIL |
ladder + explicit 192.168.1.1/32 allow @95 |
blocked | blocked | blocked | blocked | blocked | blocked | blocked | FAIL | FAIL |
| ladder minus the DHCP rules | blocked | blocked | blocked | blocked | blocked | OPEN | OPEN | OK | HTTP 200 |
| original by-hand 7-policy set | blocked | blocked | blocked | OPEN | OPEN | OPEN | OPEN | OK | HTTP 200 |
The "allow-any only" row shows Allow rules work and the port is not default-denied just by having ACLs. The by-hand row shows the enforcement mechanism is intact. The only variable that turns a working ladder into a total blackout is the pair of port-scoped UDP 67/68 allows — and note that with them present even an explicit higher-precedence Allow for the gateway does not survive.
HNS accepts them (ApplyPolicy returns success); the VFP rule set it produces drops everything. It fails closed rather than open, so it breaks jobs instead of leaking — but it makes the feature unusable.
Nothing here needs them: the endpoint is addressed by HNS IPAM, not by a DHCP client in the container. Removed in 138a737, with a regression test asserting every rule in the ladder is address-scoped and none is port-scoped.
The good news
Once those two rules are gone, the router-safe model does exactly what it claims — and this is the part that had never been validated. Blocking the whole of 192.168.0.0/16 including the container's own subnet and its default gateway does not break egress: the container still routes through 192.168.1.1 as a next hop while being unable to address it. Grafana, Incus, both Proxmox hosts, the LAN router and the ephemerd host itself are all unreachable; 1.1.1.1, 8.8.8.8, DNS-by-name via the public resolvers, and public HTTPS all work. That's a strictly better posture than the by-hand proof, which left the router and host reachable.
Mechanistic tell
vfpctrl /list-vmswitch-port enumerates the container's port on the L2Bridge switch (it errors on NAT):
Port name : E37525A9-E8DA-428A-83EF-C8F6111C07AB
Switch Friendly name : ephemerd-l2bridge
Port type : Synthetic
MAC address : 00-15-5D-6D-BE-D7
VM name : l2base-ctr@vm
Command list-vmswitch-port succeeded!
Also worth a look
setup() claimed installFirewallRules is a no-op. It isn't — firewall_windows.go programs Hyper-V firewall rules — but hyperVEgressRules is called with DefaultSubnet/defaultGateway, so on L2Bridge, where the container holds a LAN address, those rules match nothing. The per-endpoint ACLs really are the only thing enforcing. Comment corrected; the NAT-subnet scoping is left as-is since it's the NAT path's concern.
Not merging — #1 still needs an IPAM decision from you.
…opt-in)
NAT cannot software-filter Windows container egress — VFP does not engage on
an HNS NAT network, so per-endpoint Switch ACLs are inert there. This adds an
opt-in L2Bridge path that binds containers to a host NIC via a NetAdapterName
network policy, giving each container a VFP-managed vSwitch port where ACLs
actually enforce. Proven on metal (static IP); DHCP is the v1 IPAM choice.
The router-safe ACL ladder (buildL2BridgeEgressACLPolicies, a pure function)
matches the Linux end-state: block ALL of 10/8, 172.16/12, 192.168/16,
169.254/16 — whole supernets, no gateway or own-subnet carve-out — and permit
only the internet. VFP is default-DENY once any ACL is present, so the ladder
is, by precedence (lower number wins):
90 Allow DHCP (UDP 67/68, Out+In) — lease/renew survives the block
95 Allow extra-allowed CIDRs (future use; default none)
100 Block the RFC1918 + link-local supernets, Out
65500 Allow 0.0.0.0/0 Out AND In — both mandatory (the In allow keeps
TCP SYN-ACK return traffic alive; without the pair the port
default-denies everything, internet included)
DNS is set to public resolvers on the endpoint so the container never needs
the LAN router for name resolution. The rule set is static (independent of the
leased gateway/DNS) and applied at endpoint creation, before the container
starts — fail-closed, no post-lease window. Any ApplyPolicy error tears down
the endpoint and refuses the job.
NAT stays the default and is untouched: buildEgressBlockPolicies and the NAT
init/setup path are unchanged, and the new ladder is only reached when a pool
sets network.l2bridge_egress = true. The NAT block-only builder is deliberately
NOT repurposed — its whole-supernet block with no carve-out would blackhole the
NAT gateway (10.88.0.1, inside 10/8) if VFP ever engaged.
Config: network.l2bridge_egress (bool), network.host_nic (required when on),
network.public_dns (default 1.1.1.1/8.8.8.8), network.extra_allowed_destinations.
On-metal validation of this branch on mfl-win-amd64-101 (Server 2025 26100)
found the router-safe ladder blocked EVERYTHING — the internet included —
rather than only RFC1918. Bisecting the rule set on the same endpoint and the
same code path, with only these two rules varying:
blocks + allow-any + UDP 67/68 allows -> every probe fails: 1.1.1.1:443,
8.8.8.8:53, DNS, public HTTPS, and
all RFC1918 targets
blocks + allow-any -> the intended posture exactly:
Grafana 192.168.10.45:3000, Incus
192.168.12.113:8443, Proxmox
192.168.5.1/.2:8006, the LAN router
192.168.1.1:80/443 and the ephemerd
host all blocked, while 1.1.1.1:443,
8.8.8.8:53, DNS-by-name and public
HTTPS all work
Controls run alongside: the two allow-any rules alone leave everything
reachable (so Allow rules do work and the port is not default-denied by their
mere presence), and the original by-hand 7-policy set still enforces
selectively (so the mechanism is intact). The only variable that turns a
working ladder into a total blackout is the pair of port-scoped DHCP allows.
HNS accepts them — ApplyPolicy returns success — but the VFP rule set it
produces drops all traffic, and an explicit higher-precedence Allow for the
gateway does not survive it either. This fails closed (breaks jobs) rather
than open (leaks), but it makes the feature unusable.
Nothing on this path needs them: the endpoint is addressed by HNS IPAM, not by
a DHCP client inside the container. Removed, with a regression test asserting
every rule in the ladder carries an address scope and none carries a port
scope.
Also corrects two comments that no longer match reality: installFirewallRules
is not a no-op (firewall_windows.go programs Hyper-V firewall rules, though
hyperVEgressRules builds them for DefaultSubnet/defaultGateway, so on L2Bridge
they match nothing), and the DHCP-IPAM claim on initL2Bridge is wrong — HNS
rejects an L2Bridge network with no Ipams outright:
hcnCreateNetwork failed in Win32: The network does not have a subnet for
this endpoint. (0x803b0005) / ErrorCode 2151350277
so no network, endpoint or container can be created while that path is taken.
That is recorded as UNRESOLVED rather than fixed here: giving the network a
subnet works (HNS then assigns endpoint addresses itself, and the rest of the
path was verified on metal), but it means handing containers real LAN
addresses, which needs an allocation range the LAN's DHCP server will not also
hand out. That is a design decision, not a code fix.
DHCP IPAM is not available on an HNS L2Bridge network. Declaring no Ipams
fails at network creation on metal (Server 2025 26100):
hcnCreateNetwork failed in Win32: The network does not have a subnet
for this endpoint. (0x803b0005)
so no network, no endpoint, no container. The network must carry a subnet
— and once it has one, HNS assigns endpoint addresses itself, from
anywhere inside that prefix. On a real LAN that is the site DHCP server's
scope: four container runs landed on four unrelated addresses.
So ephemerd allocates the addresses. A new required key, network.ip_pool,
declares the range the operator's DHCP server never leases; each endpoint
is pinned with an explicit IpConfiguration out of that pool, released on
teardown, and endpoints left by a previous run are reserved on adopt.
There is deliberately no default pool and nothing assumes any particular
network: subnet comes from the address on host_nic, gateway from that
adapter's IPv4 default route, and either can be pinned. Missing ip_pool or
host_nic fails at config load, naming the key; a pool outside the subnet,
or one swallowing the host or the router, fails at networking init.
Also fixes container provisioning on this path. Manager.GatewayIP() drove
the per-job dind listener and the Go module proxy to the NAT gateway
10.88.0.1, which exists on no interface once the NAT network is gone —
both would fail to bind and every job would fail to provision. The
platform now reports the host's own L2Bridge address instead, and the ACL
ladder gains a single address-scoped /32 allow for it, emitted only when
dind or the module proxy is enabled. That allow opens every port the host
listens on, because a port-scoped Switch ACL blackholes the whole VFP
port; the control-plane ports are fenced back off with inbound host
firewall rules scoped to the pool, which match here only because L2Bridge
does not NAT (the reason that approach failed under NAT in #136).
The proven ACL ladder is otherwise unchanged: whole-supernet RFC1918 +
link-local blocks with no gateway or own-subnet carve-out, over the
mandatory allow-any Out+In floor, every rule address-scoped.
Still opt-in, NAT still the default, and no live node touched. Switching
an existing node needs a reboot, not a daemon restart.
It is only called from network_windows.go, so on non-Windows builds it tripped the unused linter and failed CI. The address arithmetic it feeds stays in l2bridge.go, untagged, so it keeps building and running under test on every platform.
7ab66c8 to
40d69af
Compare
`ephemerd run` constructed its own networking.Config, so a local run on a host configured for L2Bridge egress got the default instead: an HNS NAT network plus the NAT-era netsh host-firewall rules. Two consequences, both seen on a live node. The job landed UNFILTERED on NAT even though the host was deliberately configured to filter, and the netsh rules block RFC1918 host-wide -- on a host whose DNS resolver is a LAN address that severs the host's own name resolution, which showed up as image pulls failing with "no such host" and then hanging at zero bytes. The run now loads config.toml once and passes the [network] settings through workflow.Runner, matching what serve does field for field, including AllowHostAccess via needsHostAccess so dind keeps working. On L2Bridge it adopts the existing network and reserves the addresses of endpoints already on it, so it cannot hand a job an address the service is already using. A missing or malformed config is still not fatal: the run falls back to the built-in default network and, on Windows, warns that the job's egress is unfiltered. Also documents the platform split honestly in the security guide. It previously claimed Windows enforces the same block list via HCN ACLs, which is false on the NAT path -- those ACLs are a VFP construct and VFP is not engaged on a NAT switch. The guide now states plainly that the Windows default does not enforce, why no software mechanism can on that stack, and what L2Bridge requires: a wired adapter, because a Wi-Fi station cannot carry container MACs, and a reserved ip_pool with no default because any guess would collide with live DHCP leases.
Metal test 2026-08-13: deployed to a live node, broke all Windows jobs, rolled backDeployed this branch (rebased onto main, What worked
What brokeEvery Windows job fails. Containers start, JIT runners register, and then the runner never connects — GitHub reports It is not the ACLs:
Suspected causeThe explicit MAC handling is a live suspect: the endpoint carried its own MAC, distinct from the host adapter's, even though L2Bridge is supposed to rewrite container MACs to the host's. The node is itself a VM, so the upstream virtual switch may be dropping frames with an unexpected source MAC. Clean A/B on the same node, same jobs, same repo:
Also found
Before this can merge
|
The 2026-08-13 deployment died with every container unreachable: runners registered, then sat offline while their jobs stayed queued. pktmon told the whole story -- the guest was configured and ARPing for its gateway, the router's reply arrived at the NIC, and the vSwitch dropped it as "Invalid Packet" before the container port, every single time. A bisect harness (recreating the proven 2026-08-12 hand run through the daemon's containerd, one field varied per run) isolated the cause to a single fact about this HNS build (Server 2025, 26100): a Switch-rule ACL with Priority below 100 silently kills the endpoint's entire VFP dataplane. It is independent of the rule's action, direction, and address -- an irrelevant Block of a documentation IP at priority 90 reproduces the dead port, and the identical ladder with every priority at or above 100 works. HNS accepts the policy without error either way. The ladder's host-allow tier sat at priority 90 (and the extra-allow tier at 95), so any endpoint with dind enabled came up dead. The earlier "port-scoped DHCP rules blackhole the port" finding was in the same band (90/95) and was likely this trap as well. Re-tier the ladder to 100 (host allow) / 150 (extra allows) / 200 (RFC1918+link-local blocks) / 65500 (allow-any Out+In). Same precedence order, nothing below 100. A regression test pins both the constants and every emitted rule above the floor. Two more fixes from the same investigation: - Creating the L2Bridge network consumes the host NIC: it re-enumerates as "vEthernet (<name>)", so every daemon restart after the first failed plan resolution and crash-looped. Adapter lookup (address and gateway both) now tries the vEthernet-renamed form, and the unwrapped form for operators who configured the vEthernet name. - Endpoint IpConfigurations no longer set PrefixLength. HNS derives it from the network's subnet; the proven hand run and Microsoft's own sdnbridge CNI both omit it. Validated live on the win-amd64 node: with the re-tiered ladder a real CI job's runner came online in under a minute and the job executed on the L2Bridge network, ACLs enforcing (harness probe: internet up, management plane blocked). Same node, same jobs, old ladder: dead.
Root cause found, fixed, and validated live (
|
| case | shape | verdict |
|---|---|---|
| A0 | exact proof shape | PASS — no environment/version drift |
| B0 | exact production shape | FAIL — harness reproduces the bug |
| A1 | production ACL ladder only | FAIL — ladder is the killer |
| A1b | ladder minus the 192.168/16 block | FAIL — not the block scope |
| A1c | ladder minus the priority-90 rule | PASS |
| blk90 | irrelevant Block @90, nothing else changed | FAIL — priority band, not the rule |
| blk200 | same rule at @200 | PASS — third tier is fine |
| fix | allow@100 / blocks@200 / allow-any@65500 | PASS: internet up, Grafana blocked |
The commit re-tiers the ladder to 100/150/200/65500 (same precedence order, nothing below 100) and adds a regression test pinning both the constants and every emitted rule above the floor. Two more fixes from the same investigation: adapter lookup now tolerates the vEthernet (<name>) rename that creating the network causes (previously every daemon restart after first boot crash-looped), and endpoint IpConfigurations no longer set PrefixLength (HNS derives it; the proof and Microsoft's sdnbridge CNI both omit it).
Live validation on the win-amd64 node: with the fixed ladder, a real Windows CI build job's runner came online in under a minute and the job ran to success on the L2Bridge network — pinned pool address, MAC-rewrite observed working, live endpoint ladder verified at 100/200/65500. Same node and job under the old ladder: dead. Also re-observed en route: pinned IpConfigurations work fine (guest fully configured), exonerating the previous comment's prime suspect.
Still open before merge: dind-over-L2Bridge not yet exercised by a real job (the build job doesn't use docker); a new orphan-sweep race that half-deleted a provisioning job's runner dir at daemon startup (separate bug, separate fix); primary-NIC (shared) binding untested with the fixed ladder — the live node runs on a dedicated second NIC.
The orphan sweep decides "orphan" by the absence of a containerd container. But Create copies the ~200MB runner dir (job-<id>) and can then spend minutes pulling a cold Windows image before it calls NewContainer — the whole time, the job has on-disk state but no container. A sweep firing in that window (startup CleanOrphans racing the startup poll, or the periodic SweepOrphans) deleted a live job's runner dir out from under it, half- removing node.exe and leaving the runner in a "path not found" self-update loop. Observed on the win-amd64 node 2026-08-14. Track in-flight IDs on the Runtime: Create registers its ID before the copy and clears it on return; SweepOrphans unions those IDs into its keep set. Closes the window regardless of provisioning duration. CleanOrphans stays nil-keyed — it is startup-only, before any provisioning begins. Also documents the dedicated-NIC recommendation (creating the L2Bridge migrates the host IP onto a vEthernet adapter; doing that on a remote node's only NIC risks unreachability) in config.example.toml and the security guide, and corrects the security guide's wired-adapter claim: the egress ACLs rewrite container source MACs to the host NIC's, so the "Wi-Fi can't carry extra MACs" reasoning was wrong. Wi-Fi is now called out as untested rather than impossible.
An adversarial in-job probe on the win-amd64 node showed egress containment working perfectly (LAN + every management plane blocked, internet up) but `docker version` timing out. The VFP host /32 allow lets a container's packet leave its port toward the host, but the host's OWN inbound Windows Firewall default-denies it, so the per-job dind Docker API listener — bound to the host's LAN address — was never reachable. A timeout, not a refusal, confirmed the drop was at the host firewall, not the socket. Add a scoped inbound allow, opened when the Windows dind listener binds and removed when the job's server stops: dir=in action=allow protocol=TCP localip=<host> localport=<dind port> remoteip=<ip_pool>. It opens exactly that one dynamic port to exactly the container pool — a blanket host allow would expose RDP/SMB/RPC to job containers, which the strict posture must not. Plumbed through networking.Manager.OpenHostPort/CloseHostPort; no-op on NAT and on Linux/macOS (containers reach the bridge gateway directly there). A unit test pins the rule's scope so a future change can't widen it to localport=any.
…tdown Per-job dind host-port allows are removed by dind's CloseHostPort on a graceful job stop, but a hard kill (Stop-Process) skips that, and removeL2BridgeFirewallRules only deleted the control-plane rules by their computed names -- so a hard-killed job left its inbound allow behind, and they accumulated. Sweep the ephemerd-egress-l2b-hostport-* prefix via the firewall cmdlets (netsh delete-by-name has no wildcard) on every removeFirewallRules, so both shutdown and the next startup Cleanup reclaim any leaked allows. Runs regardless of whether a plan currently resolves.
The default NAT Windows path installed a Hyper-V-firewall rule set and a netsh host-firewall fallback, plus a block-only per-endpoint ACL set. All three were proven ineffective on real hardware: runhcs NAT containers register no Hyper-V VMCreator (so the rules bind to nothing), the host firewall sees NAT'd egress post-NAT (so a container-source-scoped rule matches nothing), and VFP does not engage on a NAT vSwitch (so the ACLs are inert). It was enforcement theater — a NAT node looked protected while its containers reached the whole LAN. Delete it. On the default NAT network ephemerd now installs nothing and logs plainly that container egress is NOT filtered, pointing the operator at network.l2bridge_egress (the only path that actually enforces, via HNS L2Bridge + per-endpoint VFP Switch ACLs). No security is lost — none of the removed code ever blocked anything. Removes ~975 lines: hyperVRule/hyperVEgressRules/discoverContainerVMCreators/ enableHyperVFirewallScript/removeByPrefixScript/hyperVFirewallAvailable and the WSL creator const; hostFirewallRules/installNetshFirewallRules/ removeNetshFirewallRules and their CIDR-subtraction helpers; and buildEgressBlockPolicies with its NAT applyACLPolicies branch. The L2Bridge path is untouched — the two branch cleanly on cfg.L2BridgeEgress — and the shared helpers it depends on (psQuote, powershell, netsh, winFirewallRule, egressBlockedCIDRs) are retained. Stale tests for the deleted code removed. Also corrects the docs the strip made honest: the firewall_windows.go header no longer frames the Hyper-V firewall as the "primary path," and docs/arch/windows-egress-wfp-investigation.md gains an addendum noting a host-side software path (L2Bridge VFP ACLs) was later found and shipped, so its "network-level only" conclusion applies to the NAT stack, not universally.
1 participant
Problem
Windows job containers have no enforced egress filtering. A live container
can reach Grafana, the Incus API, and both Proxmox hosts. On Linux this is
enforced by nftables (
pkg/networking/firewall_linux.go); on Windows thezero-config HNS NAT stack makes software filtering impossible — VFP does not
engage on a NAT network, so the per-endpoint Switch ACLs ephemerd already
applies are inert.
Changing the container network type to L2Bridge is the only working path:
there VFP is engaged and HNS Switch-ACLs actually enforce. (WFP, the Hyper-V
firewall on NAT, ICS, VFP-on-NAT and NDIS LWF are all settled dead ends — see
docs/and #143. Do not re-chase them.)What merges (opt-in; NAT stays the default)
An L2Bridge networking path behind
network.l2bridge_egress, defaultfalse.The NAT init/setup/ACL path is untouched — a pool only reaches L2Bridge by
opting in.
The router-safe ACL ladder (proven on metal)
VFP is default-DENY the moment any ACL is present, so a pure-Block set would
blackhole the internet too.
buildL2BridgeEgressACLPoliciesemits, asRuleType=SwitchACLs (lower Priority = higher precedence, first matchwins):
/32its own default gateway. Containers still route through the gateway as a
next hop while being unable to address it. A "block 192.168/16 except my own
subnet" carve-out would exempt the management planes, which sit on the same
/20 as the containers. Verified blocked from inside a container: Grafana,
Incus, both Proxmox hosts, the router (:80/:443), and the ephemerd host.
Verified still working:
1.1.1.1:443,8.8.8.8:53, DNS,https://example.com→ 200.
default-denies everything; the inbound one is the non-obvious half — drop
it and every outbound TCP connect dies when the SYN-ACK returns.
address scope) blackholed the entire VFP port on metal:
ApplyPolicyreturned success and nothing egressed at all, not even an explicitly allowed
destination.
138a737removed them; a regression test now fails the build ifany ladder rule carries a port scope.
rides the
0.0.0.0/0allow and never needs the blocked LAN router.Resolved: IPAM (this was the blocker)
The branch previously assumed DHCP IPAM. That is impossible. Metal, 2026-08-12:
Ipamsdeclared, HNS refuses to create the network at all:hcnCreateNetwork ... The network does not have a subnet for this endpoint (0x803b0005). No network → no endpoint → no container.IpConfigurations— four runs produced four unrelated addresses scatteredacross the declared prefix. On a real LAN that is the site DHCP server's scope.
So the network must declare a subnet, and ephemerd must own allocation.
network.ip_poolis now required wheneverl2bridge_egress = true, with nodefault. It declares the slice of the LAN that the operator's DHCP server is
configured never to lease. ephemerd allocates from it (lowest-free-first,
idempotent per container, released on teardown) and pins each endpoint with an
explicit
IpConfigurationat the LAN's own prefix length. Endpoints left behindby a previous run are reserved when an existing network is adopted.
Nothing is defaulted to any particular network, and no built-in guess exists:
l2bridge_egressfalse(NAT)host_nicip_poolsubnethost_nicgatewaypublic_dns1.1.1.1,8.8.8.8extra_allowed_destinations[](strict)Accepted
ip_poolforms: a CIDR ("192.0.2.192/27"— network and broadcastexcluded) or an inclusive range (
"192.0.2.200-192.0.2.230"). Examplesthroughout the code, config, and tests use RFC 5737 documentation addresses;
nothing encodes anyone's real LAN.
Failure modes, all fail-fast, none guessing:
ip_poolunset → config load fails, before any HNS object exists:network.ip_pool is required when network.l2bridge_egress = true: on L2Bridge, job containers are addressed on this host's own LAN rather than behind NAT, so ephemerd must be told which addresses it may hand out. Set it to a range your DHCP server is configured never to lease — either a CIDR (ip_pool = "192.0.2.192/27") or an inclusive range (ip_pool = "192.0.2.200-192.0.2.230") — sized for at least runner.max_concurrent containers. There is no default: any built-in guess would collide with live DHCP leaseshost_nicunset → config load fails, naming the key andGet-NetAdapter.network.host_nic.network.gateway.subnet, or containing this host or the gateway → fails, namingnetwork.ip_pooland the offending address.network.ip_poolandrunner.max_concurrent. Never a silent fallback to anHNS-picked address.
job refused. Fail closed: no network beats an unfiltered one.
Resolved: the
10.88.0.1dind bindingManager.GatewayIP()derived the NAT gateway from the container subnet, andthree things bind to it: the per-job dind Docker API listener
(
pkg/dind/listen_windows.go, handed to the job asDOCKER_HOST), the Gomodule proxy (
GOPROXY), and dind's fakebridgenetwork metadata. OnL2Bridge no interface holds
10.88.0.1, sonet.Listenfails outright andevery job fails to provision — the same shape of failure as the ICS
experiment.
Two changes:
platformNetworkinggainedhostAddr(). On L2Bridge the Windows platformreports the host's own adapter address, so the listeners bind to
something that exists. NAT and Linux/macOS return
""and keep the existingsubnet arithmetic — behaviour there is byte-identical.
The ladder therefore emits one address-scoped
/32allow for the host,gated on
AllowHostAccess(set fromdind.enabled || module_proxy.enabled).With neither feature on, the host stays unreachable — the strictest posture,
exactly as verified on metal.
The
/32allow opens every port the host listens on, not just ephemerd's,because a port-scoped Switch ACL blackholes the whole VFP port. The port
precision moves to the host firewall instead:
l2BridgeControlPlaneRulesemitsinbound blocks scoped
localip=<host>/localport=<control port>/remoteip=<ip_pool>. That scoping works here and did not work under NAT(#136) for exactly the documented reason — L2Bridge does not NAT, so the
container's own address is still on the packet when it reaches the host.
The NAT-oriented Hyper-V firewall rules are deliberately not installed on
this path:
hyperVEgressRulessubtracts the container subnet from every blockedrange, and on L2Bridge the container subnet is the management LAN, so the
subtraction would carve the management plane straight back out of the deny.
Tests
All pure functions, no HCN/netsh calls.
pkg/networking/l2bridge_test.goandpkg/config/network_test.gocarry no build tag, so they run on Linux CI too.Address plan / IPAM (new, platform-independent):
TestParseIPPool_Forms/_Rejects— both accepted forms, network+broadcastexclusion, /31 and /32, and every malformed spec rejected.
TestIPAllocator_AllocatesWithinPool— every address inside the pool, noduplicates, exhaustion is an error naming
network.ip_pool.TestIPAllocator_ReservesHostAndGateway,_ReleaseAndIdempotence— host androuter never handed out; retried setup reuses its address; release returns it.
TestResolveL2BridgePlan_DerivesFromHost/_ExplicitOverridesWin— subnetand gateway read off the adapter, pins honoured, derivation flagged.
TestResolveL2BridgePlan_Failures— 10 subtests, one per failure mode, eachasserting the message names the key to fix.
TestResolveL2BridgePlan_NoBuiltInNetwork— guard against re-introducing ahardcoded LAN: with no host information the plan must fail, not invent one.
ACL ladder + firewall + binding (Windows):
TestL2BridgeEgressACLPolicies_LadderShape,_NoGatewayOrSubnetCarveOut,_Precedence,_ExtraAllowed— unchanged, still passing.TestL2BridgeEgressACLPolicies_EveryRuleIsAddressScoped— the port-scoperegression guard from
138a737, now also covering the host allow. Not weakened.TestL2BridgeEgressACLPolicies_HostAllow/_NoHostAllowByDefault— the /32is above the block, address-scoped, and absent by default.
TestL2BridgeControlPlaneRules/_NoPlanNoRules— inbound, blocked, scopedto host+pool+port, names distinct from the NAT rules, silent when unscopable.
TestWindowsHostAddr_L2BridgeReportsHostIP— the provisioning regressionguard; also asserts NAT still yields
10.88.0.1.TestWindowsHostAllowIP_GatedOnAllowHostAccess.Config:
TestNetworkValidate_L2BridgeRequiresIPPool/_RequiresHostNIC/_MinimalConfig/_OptionalOverrides/_NotRequiredWhenOptedOut/_BlankStringsAreNotSet.go build ./...,go vet ./pkg/... ./cmd/...,go test ./pkg/networking/... ./pkg/config/... ./pkg/dind/...all clean on nativeGOOS=windows;GOOS=linuxandGOOS=darwincross-builds andGOOS=linux go vetclean.Honest caveats — what still needs metal
IpConfigurationson an L2Bridge endpoint is not metal-verified.The metal run let HNS auto-assign. Pinning the address is the documented HNS
approach and is what win-bridge CNI does, but the first gated run must confirm
HNS accepts the pinned address and that the container's route table comes out
right (
ipconfig, default route via the LAN gateway,pingout)./32allow tier was not run end-to-end on metal. The 2-tierladder was. Needs:
docker versionfrom inside a job (dind reachable) whilethe control ports remain refused from the same container.
Get-NetRoute -InterfaceAliasgateway derivation is untested on a liveadapter; the fallback is
network.gateway.net.InterfaceByNameon Windows uses the adapter friendly name, thesame string HNS
NetAdapterNamewants. Consistent by construction, but worthconfirming on the first run with a renamed adapter.
v0.14.0-rc.1exposes noready source-IP/MAC anti-spoof policy in a form I could apply without guessing
the schema. On L2Bridge a container is a routable LAN peer that could
source-spoof within the L2 domain; egress enforcement rests on the VFP ACLs.
Migration — required, and NOT done here
No live node was touched. In particular the kings node
(
mfl-win-amd64-101/ Proxmox VM 101) is left alone; it is processingproduction jobs on v0.1.10.
Enabling this on a host that already has the HNS NAT network requires a
reboot, not just a daemon restart. Creating/deleting an L2Bridge network
alongside a live NAT network has been observed to wedge HNS: job containers ran
with no egress at all, JIT runners registered then stayed offline,
Stop-Service ephemerdhung on graceful drain, and recovery needed a ProxmoxQMP reset.
Cutover order:
ip_poolrange on the DHCP server (or place it outside thescope) first. Duplicate-address conflicts on the LAN otherwise.
l2bridge_egress,host_nic,ip_pool(mayfly follow-up: add thekeys to the per-pool Windows schema;
host_nicandip_poolrequired whenthe pool opts in).
L2Bridge address plan resolvedlog line before uncordoning.Not in this PR
mayfly rendering of the new keys. Live validation. Anti-spoof.