multiplexd is a TCP stream multiplexer with zero-RTT stream open, deficit round-robin scheduling, flow-control with adaptive BDP estimation, transparent session resumption with unacknowledged-frame replay, and TLS 1.3 mutual authentication against a private trust store.
Table of Contents
- Features
- Architecture
- Generating Certificates
- Configuration
- Observability
- Usage
- Pre-built Binaries
- Building
- Deployment Notes
- Credits
- Low frame overhead: Fixed 8-byte frame header.
- Zero-RTT stream open (called fast-open in the protocol spec): The stream open frame carries the first data payload, allowing new streams to deliver their first bytes without a dedicated round-trip.
- No per-stream negotiation: The forward target is fixed at session configuration time.
- Up to 65535 concurrent streams per session: 32768 client-initiated and 32767 server-initiated, with a configurable per-session limit via
max_streams. Multiple parallel sessions to the same peer can be configured to scale beyond this limit.
- Transparent session resumption: During a transient transport loss, both sides enter a suspended state. The client reconnects and replays all unacknowledged frames in order over the new transport; active streams continue without application-visible disruption.
- TCP half-close: FIN-based half-close semantics are preserved end to end across the tunnel.
- Bidirectional forwarding: Forward and reverse port forwarding can run simultaneously over the same session.
- Deficit round-robin (DRR) scheduler: Outbound bandwidth is distributed fairly across active streams at byte-granularity, preventing any single stream from starving others regardless of message size. See spec.md §7 for the byte-granularity policy requirement.
- BDP estimator: Measures RTT and per-direction bandwidth from payload-driven PING/PONG cycles (a PING is sent on inbound PUSH or on acked outbound bytes, the PONG completes the cycle) and adaptively sizes the per-stream receive window and the session send cap from their own direction's estimate, so asymmetric channels get independently sized windows.
- Two-level flow control: A per-stream sliding receive window bounds per-stream in-flight data; a session-wide unacknowledged-byte cap blocks new payload.
- Memory back-pressure: Receive-window grants are linearly throttled as aggregate buffer occupancy rises between
mem_pressure.loandmem_pressure.hi, bounding memory growth under sustained load. - Multi-threaded offloading: With
ENABLE_THREADS=ON, each session runs on a dedicated thread. Parallel tunnels to the same peer distribute load across CPU cores.
- mTLS with private trust store: TLS 1.3 mutual authentication against an explicit set of trusted certificates. The
tlsobject can be omitted on trusted networks. - Hot configuration reload: Sending
SIGHUPreloads the configuration and gracefully drains existing sessions. TLS certificates, keys, and trust roots take effect for new sessions immediately. - Multi-peer routing: The
identityblock lets a node maintain simultaneous sessions with multiple named peers, with per-peer listeners and round-robin distribution across parallel tunnels. - Built-in observability: Health check, plain-text stats, and Prometheus-compatible metrics endpoints.
- Standards-compliant: ISO C11 and POSIX.1-2008, with platform-specific extensions available where supported.
multiplexd supports simultaneous forward and reverse forwarding over a single mux session:
+---------+ +------------+ +------------+ +---------+
| Local | | | | | | Forward |
| Apps |-n->| | | |-n->| Target |
+---------+ | multiplexd |-1->| multiplexd | +---------+
+---------+ | (client) | | (server) | +---------+
| Reverse |<-n-| | | |<-n-| Remote |
| Target | | | | | | Apps |
+---------+ +------------+ +------------+ +---------+
Forward — local apps connect to the client's listen address; the server forwards each mux stream to its connect target.
Reverse — remote apps connect to the server's listen address; the client forwards each mux stream to its connect target. The server can push connections to targets reachable only by the client.
The wire format is a fixed 8-byte frame header followed by an optional payload; a small set of control frames and a single hello exchange are the only shared session state. Compared to existing multiplexing protocols:
- vs. HTTP/2 (RFC 9113) / gRPC: HTTP/2 and gRPC operate on requests and responses with mandatory HPACK-compressed headers; each stream carries exactly one HTTP transaction or RPC call with method routing and per-call metadata. multiplexd operates on raw octets with no request semantics, headers, or framing above the mux layer.
- vs. SSH connection protocol (RFC 4254): SSH requires a
channel-openrequest per stream that names the service and destination, adding a round-trip before any payload can flow and imposing per-stream framing overhead; multiplexd has no channel requests, no per-stream negotiation, and no service-layer framing — the target is fixed at session configuration time.
| Feature | multiplexd | grpc-go streaming | OpenSSH port forwarding |
|---|---|---|---|
| Stream model | Raw byte sequences; fixed 8-byte header | One RPC call per stream; HPACK headers | Named channel; SSH_MSG_CHANNEL_DATA frames |
| New stream setup | SYN + first data in one flight; no per-stream round-trip | HEADERS frame; no per-stream round-trip | channel-open + channel-open-confirmation; one RTT before first payload |
| TCP half-close | FIN end-to-end transparent | END_STREAM bound to RPC lifecycle; no general TCP half-close semantics | Channel EOF maps to TCP FIN; half-close preserved |
| Session resumption | Transparent; unacknowledged frames replayed | None | None |
| Inter-stream fairness | Deficit round-robin scheduler; byte-granularity fairness | Round-robin scheduler; frame-granularity fairness | No inter-stream scheduling; systematically skewed under load |
| Flow control | Per-stream byte window + session-wide unacked-frame cap; cap blocks payload only | Per-stream byte window + connection-level byte window (both byte-based) | Per-channel byte window only |
| Adaptive window tuning | Adaptive BDP estimator | Monotonic BDP estimator | Fixed; manual tuning |
| Memory back-pressure | Linear throttle via mem_pressure.lo / mem_pressure.hi |
None | None |
| Config reload | Drains existing sessions in-process | None built-in | Re-execs the master process; existing child processes drain naturally |
| Observability | Health check, plain-text stats, Prometheus metrics | channelz (internal introspection); OpenTelemetry / Prometheus via interceptors | None |
See doc/spec.md for the full wire protocol and state-machine specification.
The design prioritizes transparent TCP semantics (full half-close support, session resumption invisible to applications), inter-stream fairness (DRR scheduling), and coexistence of bulk and interactive streams without latency inflation.
See doc/impl.md for runtime topology, send/receive paths, and maintainer-facing invariants.
multiplexd includes a built-in certificate generator for creating self-signed or CA-signed certificates. This feature requires building with OpenSSL (USE_TLS_LIBRARY=openssl).
# Generate client and server certificates with default RSA 4096 keys
multiplexd --gencerts client,server
# Specify a different server name
multiplexd --gencerts client,server --sni example.com
# Use ECDSA P-256 instead of RSA
multiplexd --gencerts client,server --keytype ecdsa --keysize 256
# Use Ed25519
multiplexd --gencerts mycert --keytype ed25519# First, generate a CA certificate
multiplexd --gencerts ca
# Then generate certificates signed by the CA, so only the CA certificate needs to be listed in authcerts for verification
multiplexd --gencerts client1,client2,client3 --sign ca| Option | Values | Description |
|---|---|---|
--gencerts |
name[,name...] | Comma-separated list of certificate names to generate |
--sni |
hostname | Server name for certificates (default: example.com) |
--sign |
name | Sign generated certificates with the named signer (uses <name>-cert.pem and <name>-key.pem) |
--keytype |
rsa, ecdsa, ed25519 | Key algorithm (default: rsa) |
--keysize |
bits | Key size in bits: RSA 4096 (default); ECDSA NIST P-224/256/384/521. Ignored when --keytype ed25519. |
Generated files: <name>-cert.pem and <name>-key.pem.
Configuration files are JSON objects. At least one of mux_listen, mux_connect, or identity.mux_connect must be present. The optional type field, when present, must be application/x-multiplexd-config; version=1. See conf_schema.json for the complete reference, including options, types, defaults, and fixed validation ranges.
Because JSON has no comments, multiplexd ignores any key whose name begins with -. The sample client.json and server.json files use this to carry template-only tuning notes such as -mux; these keys are skipped at load time and do not affect the running configuration.
{
"mux_listen": "0.0.0.0:8443",
"connect": "127.0.0.1:1080",
"tls": {
"cert": "@server-cert.pem",
"key": "@server-key.pem",
"authcerts": ["@client-cert.pem"]
}
}{
"mux_connect": "server.example.com:8443",
"listen": "127.0.0.1:1080",
"tls": {
"cert": "@client-cert.pem",
"key": "@client-key.pem",
"authcerts": ["@server-cert.pem"]
}
}Omit the tls object on trusted networks if the lower overhead outweighs the loss of protocol confidentiality, integrity, and peer authentication.
TLS: cert, key, and authcerts must all be present or all absent. The @path prefix loads a PEM file at startup; bare strings are treated as inline PEM. Mutual authentication (mTLS) is always enforced — the system CA store is never consulted.
The identity block lets a node advertise its own identity and maintain simultaneous sessions with multiple named peers.
identity.claimis this node's identity string, sent in every mux hello.identity.mux_connectis a list of mux endpoint addresses to dial. One outbound session is created per address, and the peer's identity is learned from theServerHello.identity.listenmaps peer identities to local TCP listen addresses. Each listener routes accepted connections over the session whose peer announced that identity.- All inbound mux streams, regardless of peer identity, are forwarded to the root
connecttarget. - If
identity.mux_connectoridentity.listenis configured,identity.claimis required.
Hub node — accepts inbound mux sessions from multiple clients and exposes each remote peer as a dedicated local listener:
{
"mux_listen": "0.0.0.0:8443",
"connect": "127.0.0.1:1080",
"identity": {
"claim": "hub-east",
"listen": {
"client-a": "127.0.1.1:8022",
"client-b": "127.0.1.2:8022"
}
},
"tls": {
"cert": "@hub-east-cert.pem",
"key": "@hub-east-key.pem",
"authcerts": ["@ca-cert.pem"]
}
}Spoke node — dials out to multiple hub peers and exposes each as a local listener:
{
"connect": "127.0.0.1:22",
"identity": {
"claim": "client-a",
"mux_connect": [
"east.example.com:8443",
"west.example.com:8443"
],
"listen": {
"hub-east": "127.0.0.2:1080",
"hub-west": "127.0.0.3:1080"
}
},
"tls": {
"cert": "@client-a-cert.pem",
"key": "@client-a-key.pem",
"authcerts": ["@ca-cert.pem"]
}
}Traffic arriving on 127.0.1.1:8022 is sent over the session whose peer claimed client-a; traffic on 127.0.1.2:8022 goes to client-b. On the spoke, traffic arriving on 127.0.0.2:1080 is multiplexed over the session whose peer claimed hub-east and is forwarded to the remote node's root connect target; traffic on 127.0.0.3:1080 goes via hub-west.
When ENABLE_THREADS=ON, each entry in identity.mux_connect runs on a dedicated thread. Repeating the same address opens multiple independent tunnels to that peer; new connections arriving on the matching identity.listen address are distributed across them round-robin.
{
"connect": "127.0.0.1:22",
"identity": {
"claim": "client-a",
"mux_connect": [
"east.example.com:8443",
"east.example.com:8443",
"east.example.com:8443"
],
"listen": {
"hub-east": "127.0.0.2:1080"
}
},
"tls": {
"cert": "@client-a-cert.pem",
"key": "@client-a-key.pem",
"authcerts": ["@ca-cert.pem"]
}
}This configuration opens three parallel tunnels to east.example.com:8443. Each tunnel saturates one CPU core independently, so aggregate throughput scales with the number of tunnels.
multiplexd exposes a minimal HTTP API for health checks and runtime statistics. Enable it by setting api_listen in the configuration file:
{
"api_listen": "127.0.0.1:9090"
}Returns 200 OK with an empty body when the process is running. Suitable for load-balancer or container health probes.
curl http://127.0.0.1:9090/healthyReturns a plain-text snapshot of runtime counters (no rate calculation).
Same as GET /stats, plus per-interval bandwidth rates and server CPU load derived from the time elapsed since the previous POST /stats call. Intended for periodic polling (e.g. every 10 s).
Returns metrics in Prometheus exposition format, including cumulative counters and gauges for current session/stream counts and server load.
Returns the currently active configuration as JSON, with all @path references inlined as PEM strings. Equivalent to --dump-config at runtime.
Replaces the active configuration with the JSON body of the request and performs a hot reload, identical in effect to SIGHUP with a new config file. Returns 204 No Content on success or 400 Bad Request if the body fails to parse.
# Run as server
./multiplexd -c server.json
# Run as client
./multiplexd -c client.json
# Run with colorful and very verbose output
./multiplexd -c config.json -C --loglevel 8
# Run in background and log to syslog, dropping privileges
./multiplexd -c config.json -u nobody: -d
# Dump resolved config with inlined PEM certificates to stdout
./multiplexd -c config.json --dump-configPre-built binaries are available on the Releases page. The naming convention is multiplexd[-static].<arch>-<platform>.
| Variant | Linkage | Runtime Dependencies |
|---|---|---|
-static |
Static | None — self-contained |
-android, -win32 |
Dynamic | System-provided libraries only |
| All others | Dynamic | See table below |
Non-static builds require the following runtime libraries.
| Platform | TLS Library | Install Command |
|---|---|---|
| Debian / Ubuntu | OpenSSL | apt install libev4 libssl3 |
| Alpine Linux | OpenSSL | apk add libev libssl3 |
| OpenWrt | mbedTLS | apk add libev libmbedtls |
| OpenWrt (opkg) | mbedTLS | opkg install libev libmbedtls |
Required:
- libev (>= 4.31)
Recommended (can be disabled by USE_TLS_LIBRARY=none); one of:
- OpenSSL >= 3.0
- mbedTLS >= 3.6
mkdir build && cd build
cmake ..
makecmake --build . --target check
# or rerun the registered tests directly
ctest --output-on-failureFor targeted reruns during development, pass a regex to ctest -R, for example ctest --output-on-failure -R server_test.
To build without TLS support:
cmake -DUSE_TLS_LIBRARY=none ..For common in-tree workflows, m.sh wraps configure/build/test steps from the repository root:
./m.sh genregenerates the C sources derived from JSON schemas (*.gen.c/*.gen.h). The generated files are committed to the repository, so builds without Python 3 are unaffected unless the schemas change../m.sh drebuilds a debug configuration with sanitizers enabled, then runsctest./m.sh rrebuilds a release configuration./m.sh posixrebuilds withFORCE_POSIX=ON./m.sh cremoves generated build artifacts./m.shbuilds the existing configuration
See the script for the full preset list, including clang, cross, min-size, and profiling builds.
| Option | Default | Description |
|---|---|---|
USE_TLS_LIBRARY |
auto | TLS library to use (auto, openssl, mbedtls, none) |
BUILD_STATIC |
OFF | Build a static executable (incompatible with sanitizers/systemd) |
BUILD_PIE |
OFF | Build a position-independent executable |
LINK_STATIC_LIBS |
OFF | Link against static libraries |
ENABLE_SANITIZERS |
OFF | Enable address/leak/undefined sanitizers (BUILD_STATIC=OFF) |
ENABLE_SYSTEMD |
OFF | Enable systemd state notify (BUILD_STATIC=OFF) |
ENABLE_THREADS |
OFF | Enable multi-threaded offloading |
ENABLE_ALLOC_CACHE |
ON | Enable allocation caching and pooling |
FORCE_POSIX |
OFF | Use POSIX.1 APIs only |
Run these helper scripts from the repository root:
scripts/bench.pyruns the iperf3 benchmark suite and writes a Markdown summary tobuild/bench.mdscripts/codesize.pybuilds a release configuration and writes a per-file object size report tobuild/codesize.mdscripts/gcov.pyconfigures a gcov coverage build, runs the test suite, and writesbuild/gcov.mdscripts/gen_schema.pygenerates C structs, marshal/unmarshal functions, and perfect-hash key dispatchers from JSON Schema files (also invoked by./m.sh gen)scripts/gprof.pyruns a focused gprof benchmark build and writesbuild/gprof.mdscripts/linearity_test.pyruns a bidirectional throughput/CPU linearity benchmark across increasing bandwidth limits and writesbuild/linearity_report.mdscripts/lint.pyruns clang-tidy on production sources and writesbuild/lint.mdscripts/smoke_test.pyruns an end-to-end smoke test: generates certificates, starts a server/client pair, exercises randomised TCP behavior, then verifies clean shutdown
With TLS enabled, multiplexd uses TLS 1.3 with mutual certificate authentication and a private trust store defined solely by authcerts. Each peer must present a certificate and matching private key, and a peer is accepted only if its certificate chains to a certificate explicitly configured in authcerts. Tunnel confidentiality, integrity, and peer authentication therefore depend on your provisioned certificate set, not on the system CA store, DNS names, or the public Web PKI.
Operational requirements:
authcertsshould contain only a private CA certificate or explicitly trusted peer certificates.- A CA certificate in
authcertsauthorizes every peer certificate issued by that CA; a leaf certificate entry authorizes only that specific peer. - Compromise of a trusted CA key or an endpoint private key compromises peer authentication.
*-key.pemfiles and--dump-configoutput contain private key material and must be protected accordingly.
The max_startups option (start:rate%:full format, where rate is a percentage from 0 to 100) rate-limits incoming session attempts: the first start are accepted freely; beyond that, each new attempt is rejected with probability rate% until full is reached.
api_listen exposes unauthenticated runtime statistics. Bind it to the loopback address (127.0.0.1 or ::1); never expose it to untrusted networks. A warning is logged at startup if a non-local address is configured.
There are two ways to trigger a reload: send SIGHUP to the process, or issue a PUT /config request to the API server. Both are equivalent in effect. PUT /config accepts the new configuration as a JSON body instead of reading from the config file, which makes it scriptable without touching the filesystem.
On reload, all existing sessions drain: each session stops accepting new inbound streams, completes its current streams gracefully, then reconnects with the updated configuration. There is no timeout on the drain phase — a session with long-lived streams may remain open indefinitely. Settings that can be changed at runtime include:
loglevel— applied immediately.- Live mux behavior applied to sessions before they drain: timeouts, keepalive, stream and session windows,
max_streams,max_halfopen,nodelay, andmem_pressurethresholds. - Admission controls:
max_sessionsandmax_startups. - Listener bind addresses (
listen,mux_listen,api_listen): the listener is stopped and restarted on the new address. - Forward target (
connect) and outbound addresses (mux_connect,identity.mux_connect): reconnects after drain use the new address. - TLS certificates, keys, trust roots, and TLS 1.3 cipher suites: applied to newly accepted sessions and future outbound reconnects.
Thanks to: