Migrate stellar-core Ingress to Gateway API HTTPRoute via a scalable nginx proxy by Jonathan-Eid · Pull Request #405 · stellar/supercluster

Jonathan-Eid · 2026-07-09T17:09:59Z

What

Migrate the per-mission stellar-core Ingress → Gateway API HTTPRoute, fronted by a scalable per-mission nginx proxy:

Replace the per-pod ExternalName/ClusterIP services + fat multi-rule Ingress with one nginx proxy (Deployment + Service + ConfigMap) behind a single-rule HTTPRoute. The proxy demuxes /<pod>/core|history to the pod's in-cluster DNS, so the route needs one rule regardless of node count.
Proxy replicas auto-scale ceil(nodes/64), capped by --http-proxy-replicas (default 10, floor 1), with a readiness wait before the driver routes through it.
Convert the parallel-catchup job-monitor Ingress → HTTPRoute.
Bump KubernetesClient 15 → 17 (+ typed HTTPRoute models in CSLibrary/GatewayApi.cs).
Rename internal ingress* identifiers → route* (CLI flag strings kept for Jenkinsfile compat); drop the now-dead --ingress-class.

Why

The old ExternalName-per-pod + nginx-ingress design overloaded CoreDNS (per Improve ingress handling #197) — the ingress controller re-resolving N ExternalNames + NXDOMAIN during pod startup. The proxy creates zero ExternalName services.
HTTPRoute has a hard 16-rule cap; a one-rule-per-pod route breaks past ~8 nodes. The single-rule proxy route is unaffected at any scale.

Testing

Build + unit tests (11/11) green.
Validated live on a test cluster: SimplePayment, HistoryTestnetRecentCatchup, ComplexTopology (7 nodes), SlowNodesNetwork (10), a 3-way parallel batch (survey + slow-nodes + complex, isolated per-nonce), VersionMix/DBUpgrade parallel, HistoryGenerateAndCatchup (sequential multi-network), and SimulatePubnetMixedLoad provisioning at 91 nodes → proxy auto-scaled to 2 replicas.

🤖 Generated with Claude Code

Replace the nginx Ingress (ToIngress) with an HTTPRoute (ToHttpRoute) attached to the shared traefik-gateway-private gateway. Per-pod /<pod>/core|history PathPrefix matches with a ReplacePrefixMatch URLRewrite (equivalent to the old rewrite-target). - Add typed Gateway API models (GatewayApiModels) in CSLibrary; official KubernetesClient has no Gateway API types. - Create/track/delete + orphan-sweep the HTTPRoute via the custom-object API. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Highest version before the 18.x model-constructor breaking change (parameterized ctors removed). Builds clean + tests pass with no code changes; 18/19 would need a repo-wide constructor migration, left as separate work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Traefik's Gateway provider rejects ExternalName services as HTTPRoute backends ('type ExternalName is not supported'), unlike the nginx Ingress path. Select the target pod via its statefulset.kubernetes.io/pod-name label instead. Works for both gateway and ingress; verified live on ssc-test (core /info + protocol upgrade route through the gateway). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The generated object is now an HTTPRoute, not an Ingress; name it <nonce>-stellar-core-http-route.

…e rename Replace the per-pod ClusterIP services + fat multi-rule HTTPRoute with a single per-mission nginx proxy (Deployment + Service + ConfigMap) behind a one-rule HTTPRoute. The proxy demuxes /<pod>/core|history to the pod's in-cluster DNS name, so the route needs one rule regardless of node count -- fixing the HTTPRoute 16-rule cap and the CoreDNS amplification of the old ExternalName-per-pod design. Replica count auto-scales ceil(nodes/64), capped by --http-proxy-replicas (default 10, floor 1), with a readiness wait before the driver routes through it. Rename internal ingress* identifiers -> route* (CLI flag strings kept for Jenkinsfile compatibility); drop the now-dead --ingress-class. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

socket-security · 2026-07-09T17:10:24Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	kubernetesclient@15.0.1 ⏵ 17.0.14		⁺²

View full report

Copilot

Pull request overview

Migrates Supercluster’s per-mission stellar-core HTTP exposure from per-pod Ingress + ExternalName services to a Gateway API HTTPRoute that fronts a scalable per-mission nginx proxy, and updates related mission/job-monitor routing and Kubernetes client dependencies.

Changes:

Replace per-pod service + multi-rule Ingress with an nginx proxy (ConfigMap/Deployment/Service) behind a single-rule HTTPRoute, plus a readiness wait before routing through the proxy.
Migrate parallel-catchup job-monitor from Ingress to HTTPRoute and adjust Helm values and mission set-options accordingly.
Add typed Gateway API models (GatewayApiModels) and bump KubernetesClient from 15.x to 17.x; rename ingress-centric fields/identifiers to route-centric equivalents and add --http-proxy-replicas.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/MissionParallelCatchup/parallel_catchup_helm/values.yaml	Switch job-monitor routing config from ingress fields to Gateway API parent refs + path prefix.
src/MissionParallelCatchup/parallel_catchup_helm/templates/job_monitor.yaml	Replace Ingress with `HTTPRoute` and URLRewrite-based prefix stripping.
src/FSLibrary/StellarSupercluster.fs	Create proxy resources + `HTTPRoute` and wait for proxy readiness before proceeding.
src/FSLibrary/StellarOrphanSweep.fs	Switch orphan sweeping from Ingress to `HTTPRoute` via `GenericClient`.
src/FSLibrary/StellarNetworkCfg.fs	Introduce route/proxy naming and labels; rename ingress host fields to route host fields.
src/FSLibrary/StellarNamespaceContent.fs	Track and delete `HTTPRoute` resources instead of Ingress in namespace cleanup.
src/FSLibrary/StellarMissionContext.fs	Replace ingress-related mission context fields with route fields and add `httpProxyReplicas`.
src/FSLibrary/StellarKubeSpecs.fs	Add proxy ConfigMap/Deployment/Service generation and new `HTTPRoute` creation logic.
src/FSLibrary/StellarCoreHTTP.fs	Update Host header + URL construction to use route host fields instead of ingress host fields.
src/FSLibrary/StellarCoreCfg.fs	Add constants for Gateway target and proxy config file naming/ports.
src/FSLibrary/MissionHistoryPubnetParallelCatchupV2.fs	Update Helm set-option key from `monitor.path` to `monitor.path_prefix`.
src/FSLibrary/FSLibrary.fsproj	Bump KubernetesClient dependency to 17.0.14.
src/FSLibrary.Tests/Tests.fs	Update `MissionContext` test construction for route fields + proxy replicas.
src/CSLibrary/GatewayApi.cs	Add typed POCO models for `HTTPRoute`/`HTTPRouteList` used with `GenericClient`.
src/CSLibrary/CSLibrary.csproj	Bump KubernetesClient dependency to 17.0.14.
src/App/Program.fs	Remove `--ingress-class`, rename options to route fields, and add `--http-proxy-replicas` (while keeping ingress-* flag strings).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- waitProxyReady: treat null Deployment.Status as 0 ready (status is populated asynchronously; don't NPE mission startup). - Orphan sweep: also reap legacy Ingresses (pre-migration orphans), alongside HTTPRoutes. - job_monitor HTTPRoute: omit spec.hostnames when monitor.hostname is empty (avoids invalid hostnames: [""]). - Clarify --ingress-{internal-domain,external-host,external-port} help text to describe the gateway route (flag names kept for Jenkinsfile compat). - Run fantomas over the changed files (fixes the failing format check). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Replace the nginx-ingress setup with gateway-api CRDs + traefik as the Gateway API controller + a traefik-gateway-private Gateway (matching CfgVal), so the migrated BootAndSync smoke mission (which creates an HTTPRoute) works. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…o IngressClass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…amespace) Restore the configurability the old --ingress-class flag provided: instead of hardcoding traefik-gateway-private/traefik in CfgVal, take --gateway-name and --gateway-namespace (same defaults). Lets each cluster / CI point the route at whits own gateway. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The migrated BootAndSync creates an HTTPRoute, so CI needs a gateway-api env. Traefik-in-k3s proved a version-compat maze (bundled-traefik conflict, chart value drift, missing TLSRoute CRD, CEL/k8s-version mismatches). Envoy Gateway ships matching CRDs + controller in one chart and programs cleanly. Recipe validated on local k3d (k3s 1.31, bundled traefik disabled, envoy-gateway v1.2.6, Gateway 'ssc-gateway'): BootAndSync brought up 3 nodes and synced, driver reaching them via the gateway -> proxy. Uses the new --gateway-name / --gateway-namespace flags. Also bump checkout@v2->v4, setup-dotnet@v1->v4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Use 'use ... = new GenericClient(...)' so the client is disposed and the FS0760 IDisposable warning is cleared. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The previous 'use gc = new GenericClient' disposed the shared mission kube client (GenericClient wraps it), NPEing the next API call. Use 'let ... = new': clears FS0760 via the 'new' keyword without disposing the shared client. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Jonathan-Eid and others added 6 commits July 2, 2026 14:35

Rename IngressName -> HttpRouteName; drop -ingress from the object name

f7871ce

The generated object is now an HTTPRoute, not an Ingress; name it <nonce>-stellar-core-http-route.

Convert parallel-catchup job-monitor Ingress -> HTTPRoute

ab49e35

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 9, 2026 17:10

Copilot started reviewing on behalf of Jonathan-Eid July 9, 2026 17:10 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Jonathan-Eid and others added 12 commits July 9, 2026 13:20

Drop --ingress-class from CI workflow + docs (flag removed)

a357372

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: bump k3s to v1.28.5 (traefik chart needs k8s >=1.25)

d17c2af

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CI: isolate our traefik from k3s bundled traefik (fullnameOverride, n…

1ba2221

…o IngressClass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Dispose the gateway GenericClient in the HTTPRoute orphan sweep

12290dd

Use 'use ... = new GenericClient(...)' so the client is disposed and the FS0760 IDisposable warning is cleared. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Refactor comments in NetworkCfg to improve clarity and remove redundancy

a8e1a13

Strip trailing whitespace so fantomas --check passes

52611e5

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Drop redundant comments in orphan sweep

b6d0096

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate stellar-core Ingress to Gateway API HTTPRoute via a scalable nginx proxy#405

Migrate stellar-core Ingress to Gateway API HTTPRoute via a scalable nginx proxy#405
Jonathan-Eid wants to merge 18 commits into
stellar:mainfrom
Jonathan-Eid:jonathan/ingress-to-gateway-httproute

Jonathan-Eid commented Jul 9, 2026 •

edited

Loading

Uh oh!

socket-security Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Jonathan-Eid commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing

Uh oh!

socket-security Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jonathan-Eid commented Jul 9, 2026 •

edited

Loading

socket-security Bot commented Jul 9, 2026 •

edited

Loading