kvm: encrypt the KVM live-migration data stream with QEMU-native TLS (VIR_MIGRATE_TLS)

[calvix]

Description

CloudStack's Secure Live VM Migration for KVM (CLOUDSTACK-10333 / apache#2505) secures the libvirt
control channel (qemu+tls://, port 16514) between secured hosts. However, the actual migration
data stream — the guest's memory, and for storage migration the non-shared disk — is still
transferred in plaintext over TCP (libvirt/QEMU data ports 49152+). So on the migration network the
guest RAM is exposed to an L2 eavesdropper / MITM, which is exactly the "leakage of VM state and
memory" that the original secure-migration feature set out to prevent.

This PR adds an opt-in path that encrypts and mutually-authenticates the migration data stream
using QEMU-native TLS (VIR_MIGRATE_TLS), reusing the certificates the CA framework already
provisions for the host — mirroring the existing VNC-TLS pattern. It does not change any default
behaviour: with the new setting off (the default), migration is byte-for-byte what it is today.

How it works

• New zone-scoped global setting kvm.migrate.tls (default false), on StorageManager.
• The agent advertises a host.migrate.tls capability in StartupRoutingCommand when the QEMU
  migration certificates exist under /etc/pki/qemu and qemu.conf sets migrate_tls_x509_cert_dir.
• The management server sets MigrateCommand.migrateTls only when the setting is enabled for the
  zone and both the source and destination host advertise host.migrate.tls — otherwise it
  silently falls back to the plaintext data stream, so mixed / partially-upgraded fleets keep
  migrating (VirtualMachineManagerImpl, StorageSystemDataMotionStrategy).
• MigrateKVMAsync OR-s in VIR_MIGRATE_TLS (flag 65536). The data URI stays tcp: —
  libvirt has no tls: migration URI scheme (valid schemes are tcp/rdma/unix/fd); TLS is
  negotiated over the normal tcp: connection, only the flag changes. (This was a real trap during
  development: setting the URI to tls: fails with "unsupported scheme tls in migration URI".)
• Agent provisioning: keystore-cert-import creates the /etc/pki/qemu cert set (symlinking the
  agent cert, exactly like the existing VNC set), and serviceConfig.py writes the qemu.conf
migrate_tls_x509_cert_dir + migrate_tls_x509_verify=1 keys.

Operations covered

Scenario	Behaviour
kvm.migrate.tls=false (default)	Unchanged — plaintext data stream, no behaviour change
Setting on, both hosts secured + advertise host.migrate.tls	Data stream (guest RAM + non-shared disk) encrypted + mutually authenticated via QEMU TLS
Setting on, either host not secured / older agent	Transparent fallback to plaintext (no failed migrations on mixed fleets)
Live storage migration (non-shared disk)	Same flag also covers the disk data stream

Enabling it (operator steps)

1. Upgrade the KVM agents fleet-wide (so all hosts can advertise + honour the capability).
2. Re-provision host certificates: addHost does this automatically; existing hosts via
   "Deploy Host Keys" (the CA-framework provisionCertificate), which now also lays down
   /etc/pki/qemu.
3. Set kvm.migrate.tls=true for the zone.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

N/A — agent/orchestration change, no UI (the setting appears under Global Settings).

How Has This Been Tested?

Environment: 3-node KVM + Ceph dev cluster, Ubuntu 26.04, libvirt 12.0.0, QEMU 8.2,
management 4.22.1.0, hosts already secured via the CA framework.

Unit test — MigrateKVMAsyncTest asserts the migration data URI stays tcp: when
migrateTls is enabled (guarding the "don't use a tls: URI" trap) and that the flag is set.

End-to-end on the dev cluster — live-migrated a running guest between two secured hosts with
kvm.migrate.tls=true and captured the data ports on the destination:
tcpdump -i any 'tcp portrange 49152-49215 and host <srcIP>' -w mig.pcap.

• Before (stock / setting off): strings mig.pcap | grep -c QEVM → non-zero, legible guest
  RAM strings present → plaintext.
• After (setting on, both secured): QEVM magic count = 0, no legible RAM strings, a
  TLS 1.2 handshake is visible on the data connection, and the migration still succeeds.

Non-shared disk (storage) migration — separately verified the disk (NBD) stream is also
encrypted, reproducing CloudStack's exact libvirt invocation (VIR_MIGRATE_TLS with the data URI
set to the destination IP, i.e. VIR_MIGRATE_PARAM_URI = "tcp:"+destIp as MigrateKVMAsync
does) between two secured hosts, on a domain whose local raw disk was pre-filled with a unique
marker and captured with --copy-storage-all:

• Plaintext (--tls off): the disk marker appears 43,668× in the captured NBD stream,
  alongside the QEVM RAM magic — both readable.
• TLS on: disk marker 0×, QEVM 0×, TLS records throughout, migration still succeeds.

This confirms VIR_MIGRATE_TLS encrypts both the guest-memory and the non-shared-disk (NBD)
streams under the same migrate_tls_x509_cert_dir — consistent with libvirt/QEMU's TLS-over-NBD
(requires libvirt ≥ 4.4 / QEMU ≥ 2.11; the hosts here run libvirt 12 / QEMU 8.2).

The memory and NBD connections both follow the data URI, which CloudStack sets to the destination
host's private IP (buildMigrateCommand → destination.getHost().getPrivateIpAddress()). The CA
framework always issues each host certificate with that private IP as a Subject Alternative Name
(CAManagerImpl.provisionCertificate passes privateIp/publicIp/storageIp as the cert's IP SANs), so
QEMU's mutual-TLS verification against the IP always matches — no additional hostname/SAN
configuration is required for this to work.

Mixed fleet — with one host lacking the certs/capability, the management server does not set the
flag and the migration proceeds over plaintext (no failure), confirming the fallback.

How did you try to break this feature and the system with this change?

• Setting off → verified byte-identical to current behaviour (no flag, no cert requirement).
• One host not advertising host.migrate.tls → falls back to plaintext instead of failing.
• Forced a tls: data URI → reproduced libvirt's "unsupported scheme tls" error; kept tcp: +
  flag as the correct form (now asserted by the unit test).
• Storage (non-shared disk) migration path exercised, not just RAM-only.
• Certs missing under /etc/pki/qemu → capability not advertised → host treated as unsupported.

Notes / caveats

• Opt-in, default off — zero behaviour change until an operator enables it, and only after the
  fleet is upgraded and host certs are re-provisioned.
• Requires both hosts secured (CA framework). Unsecured or older-agent hosts transparently fall
  back, so this is safe to roll out gradually.
• Reuses the existing CA-framework certificates; no new key store / CA is introduced.
• Related prior work: CLOUDSTACK-10333 / CLOUDSTACK-10333: Secure Live VM Migration for KVM apache/cloudstack#2505 (secures the control channel) — this PR extends it to
  the data stream.