refactor: extract dereference/validate pipeline from reconcile by maltesander · Pull Request #687 · stackabletech/spark-k8s-operator

maltesander · 2026-05-21T08:46:16Z

Description

added dereference and validate steps for:
- spark application
- spark connect
- spark history server
added test snapshots for smoke (app + history) and connect tests.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

Empty modules with snafu Error stubs. Wires up `mod` declarations so later commits can fill them in without further plumbing changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Defines DereferencedSparkApplication and an async dereference() function that consolidates template merging, S3 connection resolution, the early S3 TLS sanity check, and log directory resolution. Not yet wired into reconcile(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…tion Defines ValidatedSparkApplication and a sync validate() function that resolves the product image (moved from reconcile()) and runs validated_role_config. Not yet wired into reconcile(). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

reconcile() now delegates template merging, S3 resolution, TLS sanity checking, log directory resolution, image resolution, and product-config validation to the new submodules. The function body shrinks by ~70 lines and the controller's Error enum loses six variants now living in the submodules. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Aligns history with connect (which already uses controller.rs) ahead of splitting it into a controller/ submodule folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…lidation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ution Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Drift-detect the 3 ConfigMaps, ServiceAccount, and RoleBinding that persist after SparkApplication/spark-pi-s3-1 reaches phase: Succeeded. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Snapshot the full `.data` of spark-connect-server and spark-connect-executor ConfigMaps. Uses `.yaml.j2` because spark-defaults.conf and template.yaml differ between s3-use-tls dimensions (https vs http, extraJavaOptions, initContainers, minio-tls-ca volume). The spark.kubernetes.namespace line is substituted at runtime via sed/__NAMESPACE__ to survive randomised kuttl namespaces. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

adwk67 · 2026-06-01T11:54:15Z

+type Result<T, E = Error> = std::result::Result<T, E>;
+
+pub struct ValidatedSparkConnectServer {
+    pub dereferenced: DereferencedSparkConnectServer,


This departs from what we have done elsewhere (where the individual fields are included in the validated struct): also applies to the other two validate.rs files.

adwk67 · 2026-06-01T12:06:12Z

+    client: &Client,
+    scs: &v1alpha1::SparkConnectServer,
+) -> Result<DereferencedSparkConnectServer> {
+    let resolved_s3 = s3::ResolvedS3::resolve(client, scs)


This looks like fetching + validation in one step. But looking at the code it does not seem trivial to split this up.

The TLS check has been moved ito the validate step in 91bb6d5. Otherwise we can leave this as it is (until e.g. after product config has been removed) as "resolve" in this context is actually a fetch that happens to also flatten a simple enum.

adwk67 · 2026-06-01T15:51:38Z

Smoke tests:

--- PASS: kuttl (1645.54s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.0.1_s3-use-tls-false (514.35s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.7_s3-use-tls-false (531.07s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.8_s3-use-tls-true (470.61s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.8_s3-use-tls-false (582.43s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-false (314.73s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.7_s3-use-tls-true (183.37s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (403.87s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.0.1_s3-use-tls-true (145.75s)
PASS

maltesander and others added 24 commits May 20, 2026 12:49

refactor(app): scaffold dereference and validate submodules

312224b

Empty modules with snafu Error stubs. Wires up `mod` declarations so later commits can fill them in without further plumbing changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

chore(history): rename history_controller.rs to controller.rs

746a3ca

Aligns history with connect (which already uses controller.rs) ahead of splitting it into a controller/ submodule folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(history): scaffold dereference and validate submodules

aba4a07

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(history): add dereference module with log directory fetch

f343b86

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(history): add validate module with image + product-config va…

730c61c

…lidation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(history): wire reconcile through dereference + validate

8210bb1

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(connect): scaffold dereference and validate submodules

5baf8ff

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(connect): add dereference module with S3 connection fetch

82d495c

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(connect): add validate module with image + role config resol…

f8f8a67

…ution Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

refactor(connect): wire reconcile through dereference + validate

99b4a64

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

style: apply cargo fmt to wired controllers

8bfce40

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix: correct error message

32ed5f0

fix: remove obsolete errors

d731d13

fix: regenerate crate hashes

7cc6cd2

test(spark-connect): trim 10-assert to bare CR Available wait

56c0802

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(smoke): add SparkHistoryServer resource snapshot assert

4d78418

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(smoke): add SparkHistoryServer ConfigMap data snapshot

fa03bde

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(smoke): add SparkApplication resource snapshot assert

d884f61

Drift-detect the 3 ConfigMaps, ServiceAccount, and RoleBinding that persist after SparkApplication/spark-pi-s3-1 reaches phase: Succeeded. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(smoke): add SparkApplication ConfigMap data snapshot

c1af068

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

test(spark-connect): add SparkConnectServer resource snapshot assert

ecae1c4

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

maltesander self-assigned this May 21, 2026

maltesander mentioned this pull request May 21, 2026

tracking: Smooth Operator Iteration 1: Dereferencing and cluster validation stackabletech/issues#850

Open

16 tasks

maltesander added 4 commits May 21, 2026 10:47

docs: adapt changelog

3b68fae

fix(smoke): improve diff output

e173fac

fix(tests): fix test asserts for vector aggregator

5f708ba

fix(connect): add vector aggregator fixes

47ec257

adwk67 self-requested a review June 1, 2026 09:41

adwk67 added this to Stackable Engineering Jun 1, 2026

adwk67 moved this to Development: In Review in Stackable Engineering Jun 1, 2026

adwk67 reviewed Jun 1, 2026

View reviewed changes

adwk67 added 3 commits June 1, 2026 16:42

move TLS check into the validate step

91bb6d5

fix merge conflict

1f5b3a7

thread dereferenced fields through the validated structs

043c0a0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor: extract dereference/validate pipeline from reconcile#687

refactor: extract dereference/validate pipeline from reconcile#687
maltesander wants to merge 31 commits into
mainfrom
feat/dereference-validate-steps

maltesander commented May 21, 2026 •

edited by adwk67

Loading

Uh oh!

adwk67 Jun 1, 2026

Uh oh!

adwk67 Jun 1, 2026

Uh oh!

adwk67 Jun 1, 2026

Uh oh!

adwk67 Jun 1, 2026 •

edited

Loading

Uh oh!

adwk67 commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

maltesander commented May 21, 2026 • edited by adwk67 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

adwk67 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

adwk67 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

adwk67 Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

adwk67 Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adwk67 commented Jun 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

maltesander commented May 21, 2026 •

edited by adwk67

Loading

adwk67 Jun 1, 2026 •

edited

Loading