Fix SBR disconnected CI and add 4.22 disconnected SBR jobs - #82791
Fix SBR disconnected CI and add 4.22 disconnected SBR jobs#82791ugreener wants to merge 1 commit into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR updates disconnected Medik8s AWS NFS tests, adds 4.22 SBR jobs, centralizes UBI image mirroring, and provisions concurrent NFS-backed volumes. ChangesMedik8s disconnected NFS testing
Estimated code review effort: 2 (Simple) | ~15 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
openshift-ci
Bot
added
the
approved
f4e35c9 to
ae5706e
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
ae5706e to
f162337
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ugreener The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@ugreener: |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
f162337 to
8333a9e
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
8333a9e to
ae5d490
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
ugreener
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/medik8s/sbr/nfs-bastion/medik8s-sbr-nfs-bastion-commands.sh`:
- Around line 182-185: Guard the PV diagnostic pipeline using the existing
oc/grep command so an empty nfs-sbr match does not trigger failure under set
-euo pipefail. Preserve failures from actual command errors while allowing grep
to return no matches, for example by applying the established benign-empty-match
guard to grep.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 3e69f2c8-eca1-4c4a-b856-1e47a8308391
⛔ Files ignored due to path filters (2)
ci-operator/jobs/medik8s/system-tests/medik8s-system-tests-main-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/medik8s/system-tests/medik8s-system-tests-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (4)
ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.21-disconnected.yamlci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-disconnected.yamlci-operator/step-registry/medik8s/disconnected-catalogsource/medik8s-disconnected-catalogsource-commands.shci-operator/step-registry/medik8s/sbr/nfs-bastion/medik8s-sbr-nfs-bastion-commands.sh
🚧 Files skipped from review as they are similar to previous changes (3)
- ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.21-disconnected.yaml
- ci-operator/step-registry/medik8s/disconnected-catalogsource/medik8s-disconnected-catalogsource-commands.sh
- ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-disconnected.yaml
| oc get pv -l '!kubernetes.io/cluster' --sort-by=.metadata.name | grep nfs-sbr | ||
|
|
||
| # Wait for ITMS to propagate to all nodes by actively verifying the image is pullable. | ||
| # The previous approach (watching MCP rendered config names) is unreliable: ITMS may | ||
| # update CRI-O config via a path that does not change the rendered MachineConfig name. | ||
| # Active verification is the only reliable signal that nodes have picked up the mirror. | ||
| log "Waiting for ubi8/ubi-minimal to be pullable via ITMS mirror (up to 12m)..." | ||
| UBI_PULLABLE=false | ||
| for i in $(seq 1 24); do | ||
| sleep 30 | ||
| oc delete pod sbr-ubi-preflight --ignore-not-found --wait=false 2>/dev/null || true | ||
| POD_PHASE=$(oc run sbr-ubi-preflight \ | ||
| --image=registry.access.redhat.com/ubi8/ubi-minimal:latest \ | ||
| --restart=Never \ | ||
| --command -- sh -c 'echo PREFLIGHT_OK' 2>/dev/null && \ | ||
| timeout 30 bash -c 'until [[ $(oc get pod sbr-ubi-preflight -o jsonpath="{.status.phase}" 2>/dev/null) =~ ^(Succeeded|Failed)$ ]]; do sleep 2; done; oc get pod sbr-ubi-preflight -o jsonpath="{.status.phase}"' \ | ||
| 2>/dev/null || true) | ||
| if [[ "$POD_PHASE" == "Succeeded" ]]; then | ||
| log " ubi8/ubi-minimal pullable after $((i * 30))s" | ||
| UBI_PULLABLE=true | ||
| oc delete pod sbr-ubi-preflight --ignore-not-found 2>/dev/null || true | ||
| break | ||
| fi | ||
| log " attempt ${i}/24: image not yet pullable (phase=${POD_PHASE:-pending}), retrying in 30s..." | ||
| oc delete pod sbr-ubi-preflight --ignore-not-found --wait=false 2>/dev/null || true | ||
| done | ||
|
|
||
| if [[ "${UBI_PULLABLE}" != "true" ]]; then | ||
| log "WARNING: ubi8/ubi-minimal not pullable after 12m — ITMS may not have propagated." | ||
| log " SBR init Job may fail; device files are pre-created so it will retry quickly." | ||
| oc get mcp 2>/dev/null || true | ||
| fi | ||
| # UBI image mirroring is handled by medik8s-disconnected-catalogsource step | ||
| # (additionalImages + ITMS). No manual mirror needed here. |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Inspect script header flags and full context around the PV listing line.
set -euo pipefail
fd -a medik8s-sbr-nfs-bastion-commands.sh ci-operator/step-registry | while IFS= read -r f; do
echo "=== $f (head, flags) ==="
sed -n '1,10p' "$f"
echo "=== $f (around line 182) ==="
grep -n "kubernetes.io/cluster\|pipefail\|grep nfs-sbr" "$f"
doneRepository: openshift/release
Length of output: 962
🏁 Script executed:
#!/bin/bash
set -euo pipefail
f=ci-operator/step-registry/medik8s/sbr/nfs-bastion/medik8s-sbr-nfs-bastion-commands.sh
echo "=== relevant line ranges ==="
sed -n '160,190p' "$f" | cat -n
echo "=== PV creation/storageclass/PV-related occurrences ==="
rg -n "PersistentVolume|storageClassName|CreatePersistentVolume|nfs-sbr|kubernetes\.io/cluster|-l '!kubernetes\.io/cluster'\s*\|" "$f"
echo "=== related step config references ==="
rg -n "medik8s-sbr-nfs-bastion|medik8s-disconnected-catalogsource" ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-disconnected.yaml
echo "=== deterministic bash/pipeline behavior probe for empty grep with pipefail ==="
python3 - <<'PY'
import subprocess, shlex
cases = (
("empty input without pipefail", "set -eu; true | grep xyz"),
("empty input with pipefail", "set -euo pipefail; true | grep xyz"),
("non-empty input with pipefail", "set -euo pipefail; true | grep -q xyz"),
)
for name, cmd in cases:
p = subprocess.run(["bash","-c", cmd], text=False, capture_output=True)
print(f"{name}: exit={p.returncode}, stdout={list(p.stdout)}, stderr={list(p.stderr)}")
PYRepository: openshift/release
Length of output: 2469
🏁 Script executed:
#!/bin/bash
set -euo pipefail
f=ci-operator/step-registry/medik8s/sbr/nfs-bastion/medik8s-sbr-nfs-bastion-commands.sh
echo "=== YAML manifest files in the step registry directory ==="
fd -e yaml -e yml . ci-operator/step-registry/medik8s/sbr/nfs-bastion | sort
echo "=== script line count and PV-related lines including manifests ==="
wc -l "$f"
rg -n -C 2 'CreatePersistentVolume|oc create|cat <<EOF|PersistentVolume|PersistentVolumeClaim|StorageClass|storageClassName|kubernetes\.io/cluster|kubernetes\.io/.*\.gce|openshift\.io/.*\.gce|gcePersistentDisk|nfspath|fileSystemType' "$f"
echo "=== bash pipeline edge-case probe for empty grep with different set options ==="
python3 - <<'PY'
import subprocess
for opts in ("set -eu;", "set -euo pipefail;"):
cmd = opts + "true | grep xyz"
r = subprocess.run(["bash", "-c", cmd], text=False, capture_output=True)
print(f"{opts.rstrip(';')}: exit={r.returncode}")
PYRepository: openshift/release
Length of output: 2587
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== all PVC/PV manifests in step registry files matching medik8s nfs-bastion ==="
fd -a 'medik8s.*nfs-bastion.*' ci-operator/step-registry | while IFS= read -r f; do
echo "=== $f ==="
rg -n 'kind: (PersistentVolume|PersistentVolumeClaim|StorageClass)|name: nfs-sbr|kubernetes\.io/cluster|storageClassName:' "$f"
done
echo "=== all PVC creation/parsing in medik8s sbr nfs-bastion commands ==="
f=ci-operator/step-registry/medik8s/sbr/nfs-bastion/medik8s-sbr-nfs-bastion-commands.sh
rg -n -C 3 'PersistentVolumeClaim|pvc|PV|PVs|oc create|oc apply|apply -f - <<EOF|sort_by=' "$f"Repository: openshift/release
Length of output: 767
Guard the PV diagnostic pipeline against benign empty matches.
set -euo pipefail is enabled, so an empty grep nfs-sbr result can make a successful PV-creation step abort. Capture or guard this diagnostic; use || true or grep ... || true so the step only fails for actual command errors, not for missing matches.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/medik8s/sbr/nfs-bastion/medik8s-sbr-nfs-bastion-commands.sh`
around lines 182 - 185, Guard the PV diagnostic pipeline using the existing
oc/grep command so an empty nfs-sbr match does not trigger failure under set
-euo pipefail. Preserve failures from actual command errors while allowing grep
to return no matches, for example by applying the established benign-empty-match
guard to grep.
ae5d490 to
4b8d095
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
4b8d095 to
da7f1e2
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
da7f1e2 to
28d4c1f
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@ugreener: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
The SBR disconnected periodic job (4.21-disconnected) has been failing on all 6 runs with 'panic: WORKLOAD_IMAGE environment variable must be set'. Root cause: the e2e-test commands block was missing the WORKLOAD_IMAGE export from SHARED_DIR, which the FAR disconnected job in the same repo has. Also missing ENABLE_IDMS and EXTRACT_MANIFEST_INCLUDED env vars required for disconnected image mirroring. 4.21-disconnected fixes: - Add WORKLOAD_IMAGE export from SHARED_DIR/workload_image - Add ENABLE_IDMS and EXTRACT_MANIFEST_INCLUDED env vars - Add 3h timeout matching FAR disconnected pattern 4.22-disconnected additions: - New periodic: e2e-sbr-weekly-aws-disconnected-nfs (Wednesday 16:00) - New presubmit: e2e-sbr-aws-disconnected-nfs (optional, on-demand) - Both use medik8s-sbr-nfs-bastion for NFS storage - Both include disconnected test exclusions for 10 test IDs Jira: RHWA-1039 Co-Authored-By: Claude <noreply@anthropic.com>
28d4c1f to
2dffdd6
Compare
|
/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-sbr-aws-disconnected-nfs |
|
@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
1 participant
Summary
ubi9/ubi-minimal:9.8toadditionalImagesinmedik8s-disconnected-catalogsourceand add ITMS rule so the operator's init Job can pull on disconnectedmedik8s-sbr-nfs-bastion(now handled centrally bymedik8s-disconnected-catalogsource)Changes
WORKLOAD_IMAGEexport,ENABLE_IDMS/EXTRACT_MANIFEST_INCLUDEDenv vars, 3h timeout, observers. Remove allECO_TEST_LABELSexclusions.e2e-sbr-aws-disconnected-nfspresubmit ande2e-sbr-weekly-aws-disconnected-nfsperiodic (Wednesday 16:00 UTC). No test exclusions.ubi9/ubi-minimal:9.8toadditionalImagesand ITMS for the SBR operator init Job image.make jobsJira: RHWA-1039