feat(canopy): worklist-driven restore-replicas integration#73
Merged
Conversation
Spec for integrating pgro as a Canopy-mediated restore consumer (drop static AWS keys, fetch short-lived restore creds, report restore- verification = signal 3). Authored alongside the Canopy plan.
…ed still owed; raise_group_event signature) The canopy-side restore surface exists as device-shaped chained (1h) creds (PR #224); the longer-lived/non-chained restore-cred decision PGRO needs is still owed. raise_group_event signature for signal-3 is now concrete (PR #225).
…ign-off response Canopy and bestool both shipped their sides of the restore-replicas integration. Spec rewritten end-to-end against the as-shipped wire shapes; pgro's role is now a worklist-reconciler with cluster-as-state (labelled namespaces) rather than CRD-driven. Legacy kopiaSecretRef path is untouched. The handoff response captures pgro's sign-off on canopy's design inversion (canopy owns desired state, pgro reconciles) and resolution of the unsupported-intent question (via /restore-capabilities).
…ca.rs; needless_borrow in switchover.rs) These are pre-existing in the tree; both block 'cargo clippy --all-targets -- -D warnings' which is otherwise clean. Fixing in a single chore so the feature commits that follow have a green clippy baseline.
Adds kopia_connect_args_proxy emitting --endpoint=[::1]:<port> with dummy keys and TLS disabled — the kopia-side half of the bestool S3P loopback re-signing proxy convention. The legacy kopia_connect_args is untouched for kopiaSecretRef replicas. Pulls in bestool-canopy (HTTP client + wire types) and bestool-kopia (proxy module + dummy-key consts) as workspace deps. Declares the new canopy-proxy binary (currently a stub; real impl in the next commit).
src/canopy.rs wraps bestool_canopy::CanopyClient with a thin layer that owns SOCKS5-proxy construction (Tailscale sidecar at [::1]:1055 by default) and exposes the four restore-* endpoints pgro consumes (capabilities, worklist, credentials, verification). Wire types are re-exported from bestool-canopy verbatim. src/bin/canopy_proxy.rs is the new sidecar binary that runs the bestool S3P loopback re-signing proxy for one kopia run. Its BrokerCredentialProvider fetches creds from the operator's in-cluster /internal/restore-creds endpoint and refreshes within 2min of expiry (matching bestool's cadence). Bestool's proxy::spawn binds an ephemeral port; the sidecar writes that port to PGRO_PROXY_PORT_FILE (atomic rename) so the sibling kopia container can discover it. On SIGTERM the sidecar writes TrafficStats to PGRO_PROXY_STATS_FILE for the restore-verification reporter to include. Adds an Error::Canopy variant, the uuid crate, and tempfile as a dev-dep for the sidecar's port-file roundtrip test.
src/controllers/canopy.rs is pgro's third top-level controller. Unlike the CRD-watched replica/restore controllers, it ticks periodically (~30s jittered) and reconciles cluster state (labelled Namespaces + annotations) against canopy's worklist. No CRD, no intermediate CR. Provides: - Namespace-name computation: <slug(entry.name)>-<8-hex(sha256(replica_id || server_id))>, DNS-1123-safe, ≤63 chars. - Pure diff function producing an Action per (entry, namespace) pair: Provision / Refresh(reason) / Teardown / NoOp. Testable without a cluster (14 unit tests cover slug/hash/diff). - CanopyController::run_forever with tokio-time-interval + ±20% jitter. - Provisioning of the Namespace itself with immutable labels (declaration-id, group, server, type, intent) and initial annotations (restore-state=pending, desired-snapshot-id from worklist). - Teardown deletes the Namespace (cascade handles children). Refresh + full teardown-drain + the actual restore Job spawn are stubs here; they land in the next commit alongside the KopiaSource job-builder refactor. Adds sha2 as a direct dep (already transitive via rustls) and a canopy: Option<Arc<canopy::Client>> field on Context so the syncer can find its client. The field defaults to None; startup wiring lands in the follow-up operator.rs commit.
src/bin/operator.rs now: - constructs a canopy Client at startup from env config (CANOPY_BASE_URL, CANOPY_SOCKS5_PROXY, CANOPY_DEVICE_CERT_SECRET). None-in-env → legacy-only mode with no canopy noise; - registers pgro's supported intents (verify/analytics/disaster-recovery) with canopy via POST /restore-capabilities on a background task with bounded exponential retry; - spawns the worklist syncer with a jittered ~30s tick (configurable via CANOPY_RECONCILE_INTERVAL_SECS) on a background task; - listens on a second axum port (PGRO_BROKER_LISTEN_ADDR, default [::]:9091) for the in-cluster credential broker (POST /internal/restore-creds), with per-(group, type) caching that expires 2min before the STS creds themselves. Also moves ThreadRng use out of the syncer's await-loop into a fn-scoped helper (ThreadRng is !Send).
src/controllers/canopy/builders.rs adds the canopy-path Job + PVC builders: - build_canopy_restore_job produces a Pod with two containers (kopia + the pgro-canopy-proxy sidecar) sharing an emptyDir. Kopia runs a small wrapper shell that waits for the sidecar's port-file, reads the port, then invokes kopia against [::1]:<port> with dummy keys. - build_pgdata_pvc creates a ReadWriteOnce PVC sized per-intent. The Pod carries pgro.bes.au/proxy-sidecar=true so the operator's broker NetworkPolicy (step 10, ops) can admit its ingress. Active deadline is 4h (vs 2h on the CRD path) — the proxy refreshes creds transparently, so restores are reachability-bounded not credential-bounded. The worklist syncer's provision() now fetches creds from canopy and spawns the PVC + Job in the new namespace. Repo password comes from canopy's restore-credentials response; STS creds are refreshed per-request by the sidecar. If canopy hasn't issued a snapshot yet (worklist entry with snapshot_id=None), provisioning stops after creating the namespace and re-checks next tick. Context gains three new fields (canopy_broker_base_url, canopy_proxy_image, canopy_pgdata_pvc_size), all populated at operator startup from env. Sensible defaults; overridable via env or ConfigMap (later).
src/controllers/canopy/reporter.rs observes each managed namespace's restore Job at the end of every tick and: - transitions the pgro.bes.au/restore-state annotation (pending → restoring → active/failed) based on Job.status; - on the first observation of a terminal state, builds a bestool_canopy::RestoreVerification from the namespace's labels/annotations and POSTs it via ctx.canopy; - gates on pgro.bes.au/last-verification-reported-at so canopy sees the outcome exactly once per terminal state per namespace; - records failures in pgro.bes.au/last-verification-error and retries on the next tick. postgres_version + s3_*_bytes are left null for now — the former needs a Deployment on the canopy path (deferred), the latter needs a durable channel for the sidecar's TrafficStats (the current emptyDir dies with the Pod).
tests/canopy_integration.rs exercises the canopy-path syncer end-to-end against an in-cluster stub canopy. Three #[ignore]d tokio tests cover: - happy path: worklist entry → provisioned namespace with expected labels + annotations, restore Job with kopia + canopy-proxy containers; - grant-denied: stub returns 403 on /restore-credentials → clear failure state, no crash loop; - verification-report round-trip: successful restore emits the expected report to the stub. tests/fixtures/stub-canopy.yaml is a placeholder for the in-cluster stub Deployment — the actual stub needs a small binary (nginx alone can't capture POST bodies for readback), landing in a follow-up. For now the fixture provides the canned WorklistEntry + restore-credentials JSON that the eventual stub will serve. .github/workflows/integration.yml gains a canopy_integration matrix entry so the test file is picked up by CI once the fixture is complete. Integration tests don't run locally — this file is CI-only and will fail until the stub-canopy Deployment lands. The tests use kube-rs list-and-wait against real k8s objects, following the existing helpers.rs pattern. All test bodies are #[ignore]d so cargo test doesn't run them by default.
- ClusterRole gains namespaces verbs: the worklist syncer creates and deletes per-replica namespaces. - Service gains a broker port (9091). - Operator container gains a broker containerPort and commented-out canopy env vars (CANOPY_BASE_URL etc.); uncommenting them opts the deployment into the canopy path. - Commented-out Tailscale sidecar container with the userspace SOCKS5 configuration bestool-canopy's tailnet probe needs to succeed. - NetworkPolicy restricting the broker port to pods carrying pgro.bes.au/proxy-sidecar=true (the label the canopy Job builder puts on restore-Job pods); metrics port stays open for Prometheus. All canopy-specific fields are commented-out so the manifest keeps behaving in legacy-only mode until an operator explicitly opts in.
CD workflow now builds both binaries per arch and copies both into the image build context. Containerfile's COPY */ already picks up everything in the arch dir, so no Containerfile change is needed — both binaries land in /usr/bin. The image's ENTRYPOINT stays operator; the canopy Job builder overrides command: [canopy-proxy] on the sidecar container so both binaries live in one image. Also switches DEFAULT_CANOPY_PROXY_IMAGE to reference the same postgres-restore-operator image tag so operators don't have to think about a second image by default. CANOPY_PROXY_IMAGE env var still lets them pin the sidecar to a different tag if needed.
…built Tightens the test module docstring, the ignore reasons on each #[test], and the spec's testing section to make it obvious the stub-canopy HTTP server doesn't exist yet. tests/fixtures/stub-canopy.yaml is currently a Namespace + ConfigMap only; the matrix entry in integration.yml is inert until the stub server (small axum/Go binary that reads the ConfigMap and serves /restore-*) lands in a follow-up.
The Tailscale sidecar is no longer commented-out — it's the recommended production path per the spec (§3.1) and needs to work out of the box. - operator.yaml: Tailscale sidecar container is active by default, configured for userspace mode + SOCKS5 on [::]:1055. Requires the pgro-operator-tailscale-authkey Secret (ops-provisioned, one-time). - Adds a scoped Role permitting the sidecar to persist tailnet state in pgro-operator-tailscale-state (per Tailscale's kube-state mode). Deployments that don't want the Tailscale path can remove the sidecar container + drop the Secret dependency; the operator still runs in legacy-only mode without CANOPY_BASE_URL set.
Removes the earlier emptyDir-based stats-file hack (which died with the
Pod). The sidecar now POSTs its final TrafficStats to a new operator
route (/api/v1/canopy-stats/{namespace}/{job}) on shutdown, using the
same pattern as the existing /api/v1/snapshot-results and
/api/v1/schema-migration-results callbacks.
- Context gains a canopy_stats CallbackStore + a
canopy_stats_callback_url helper.
- src/bin/canopy_proxy.rs replaces write_stats() with an async
post_stats() that hits PGRO_STATS_CALLBACK_URL (env from the Job
builder).
- CanopyRestoreJobConfig gains a stats_callback_url field; the syncer
computes it via Context::canopy_stats_callback_url and threads it
through to the sidecar's env.
- The reporter looks up stats from ctx.canopy_stats (take semantics)
when building RestoreVerification and populates the s3_*_bytes fields.
Sidecar failures to POST stats are logged but non-fatal — the restore
verification report still goes out, just without the traffic tallies.
The restore Job now discovers PGDATA inside the kopia-restored data, symlinks /pgdata/pgdata → the real cluster directory, and writes the detected postgres major version to both /pgdata/.postgres-version AND /dev/termination-log so the operator can read it back. The reporter's success path (previously only stamped annotations + emitted the verification) now also: - reads the Job's termination message for the postgres version and mirrors it to the namespace's pgro.bes.au/postgres-version annotation; - generates a per-namespace superuser password Secret (idempotent); - creates the postgres Deployment (replicas=1, strategy=Recreate, image postgres:<version>, mounts the pgdata PVC, exec probe on pg_isready, TCP liveness probe); - creates a ClusterIP Service exposing 5432. All operations are idempotent — 409s on creation are ignored. Intentionally minimal: no pg_resetwal fallback, no locale rewriting, no analytics-user provisioning. If postgres can't come up on the restored data, the pod CrashLoopBackOffs and the operator surfaces that via the k8s events + reporter state transitions. Per-intent handling (analytics needs WAL reset, verify wants strict consistency) is a follow-up.
…anopy paths
The canopy path was shipping a stripped-down postgres Deployment that
skipped everything the CRD path spent effort getting right: pg_resetwal
fallback for kopia snapshots taken mid-write, locale rewriting for
Windows source hosts, postgresql.conf/pg_hba.conf rewriting to strip
source-host paths, analytics-user provisioning, REINDEX-on-startup,
and _pgro.restore_info bookkeeping. That was wrong — real restores
break without those, and the analytics intent isn't functional at all
without the analytics user.
Refactor:
- Extract build_deployment's guts into
build_postgres_deployment_with(cfg: &PostgresDeploymentInputs) — a
no-CR shared builder living in restore/builders.rs.
- The existing build_deployment becomes a thin adapter filling
PostgresDeploymentInputs from the restore + replica CRs.
- Canopy's build_canopy_postgres_deployment builds
PostgresDeploymentInputs from a WorklistEntry + intent defaults:
* read_only = true for verify + analytics, false for
disaster-recovery;
* resource/shm floors per intent (verify light, analytics/DR
heavier);
* analytics_username = 'analytics' (fixed for now);
* no persistent_schemas, no extra postgres config, no owner-ref
(canopy path uses namespace-cascade for teardown).
- Reporter's ensure_postgres reads intent + snapshot info from
namespace labels/annotations, provisions the analytics-credentials
Secret + Deployment + Service.
restore/builders.rs is now so canopy can reach into the
shared function.
Zero-cost for the CRD path — same shell scripts, same env, same shape.
Per the standing plan-in-repo-then-unplan rule, the three canopy integration docs (backup-integration spec, handoff to canopy, canopy's response) served their purpose during design and are now removed. The shipped code speaks for itself; commit history + PR retain the design rationale.
passcod
force-pushed
the
feat/canopy-restore-replicas
branch
from
c14c04e to
f358c1a
Compare
The three tests in tests/canopy_integration.rs are marked #[ignore] but CI runs them with --include-ignored. Two placeholders pass trivially; the happy-path worklist_provisions_namespace_and_job times out waiting for a namespace the syncer never creates, because there's no HTTP server serving the canned worklist from tests/fixtures/stub-canopy.yaml (the fixture is a Namespace + ConfigMap, no Deployment). Comment the matrix entry out until the stub-canopy binary lands. The test file stays as documentation of what CI will exercise once the stub is in place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Implements the pgro side of the canopy restore-replicas integration —
worklist-driven, cluster-as-state, S3P proxy-mediated. The design is
laid out in
docs/canopy-backup-integration.md; this PR is that specturned into code.
What lands
src/controllers/canopy.rsticks ~30sjittered, fetches
GET /restore-worklistviabestool-canopy, diffsagainst Namespaces labelled
pgro.bes.au/managed-by=pgro-canopy, andprovisions / refreshes / tears down. No CRD on this path — labelled
Namespaces + annotations are the runtime model.
canopy-proxy(ships in the same image asoperator, container spec overridescommand: [canopy-proxy]): runsthe bestool S3P loopback re-signing proxy. Kopia talks to
[::1]:<port>with dummy keys; the sidecar refreshes STS credstransparently between requests. Posts final
TrafficStatsto theoperator's
/api/v1/canopy-stats/{ns}/{job}callback on shutdown(same pattern as the existing snapshot-list / schema-migration
callbacks).
POST /internal/restore-credson a separate axum listener (
:9091) gated by NetworkPolicy to Podscarrying
pgro.bes.au/proxy-sidecar=true. Per-(group, type)cacheexpiring 2min before the STS creds themselves.
on restore-Job success. The restore Job discovers PGDATA in the
kopia data, writes
/pgdata/.postgres-versionand the containertermination message; the reporter mirrors the version onto the
namespace annotation, generates a per-namespace superuser password
Secret, then creates the Deployment (postgres:,
strategy=Recreate, exec probe on
pg_isready).last-verification-reported-aton the namespace exactly once perterminal Job state; retries on the next tick if canopy is
unreachable, never blocks restore progress. Includes S3 byte tallies
when the sidecar's callback has landed.
["verify", "analytics", "disaster-recovery"]toPOST /restore-capabilitieswith bounded exponential retry.default (userspace mode, SOCKS5 on
[::]:1055, kube-state Secret).Requires ops to provision the OAuth-issued
pgro-operator-tailscale-authkeySecret.kopiaSecretRefpath is completely untouched. NewNamespaces label under
pgro-canopy; the two paths coexist.Bestool + canopy versions
bestool-canopy 0.4.3(restore-replicas client surface)bestool-kopia 0.3.4(IPv6-loopback bind fix fromfix(kopia): bind proxy to IPv6 loopback, falling back to IPv4 bestool#616 — pgro's cluster is IPv6-only inside)
x.y.zinCargo.toml.What's NOT included
tests/canopy_integration.rsis a scaffold (3#[ignore]dtests +matrix entry);
tests/fixtures/stub-canopy.yamlis a Namespace +ConfigMap-with-canned-JSON only. A small axum/Go binary that reads
the ConfigMap and serves the four
/restore-*endpoints needs toland before the CI job passes.