[release-4.22] OCPBUGS-94186: Add retry for transient API errors to prevent Degraded blips by sg00dwin · Pull Request #1181 · openshift/console-operator

sg00dwin · 2026-06-30T19:06:52Z

Summary

Manual cherry-pick of #1164 to release-4.22.

Automated cherry-pick failed due to a merge conflict in pkg/console/operator/sync_v400.go (the ApplyConfigMap call site). Resolved manually — took the incoming retry wrapper.

Original PR

#1164

Test plan

Build clean
All unit tests pass
Conflict resolved: single site in sync_v400.go where ApplyConfigMap needed retry wrapper

blips Wrap all API write operations (Apply, Create, Update, Delete) with retry logic to absorb transient errors (conflicts, timeouts, connection refused) during upgrades before they reach status reporting. Without retry, a single transient failure sets Degraded=True for ~1 minute until the next sync resolves it, causing CI test failures. Retry parameters: 3 attempts, 500ms backoff with 2x factor (~3.5s max). Permanent errors (Forbidden, Invalid, AlreadyExists) are not retried. Assisted by Claude Code (Opus 4.6)

…de the retry closure before calling Update/UpdateStatus. Each retry attempt now uses the current resourceVersion instead of the original stale copy. - SyncConsoleConfig: re-fetches consoleConfig before UpdateStatus - SyncServiceCAConfigMap: re-fetches configmap and re-applies metadata before Update - SyncTrustedCAConfigMap: same pattern - ApplyCLIDownloads: re-fetches CLI downloads resource, re-applies spec and metadata before Update Assisted by: Claude (Opus 4.6)

- Guard deleteStorageVersionMigration against NotFound errors - Remove one-line retryOnTransientError wrapper, call util directly - Remove misleading nil-error test case from IsRetryableError Assisted-by: Claude (Opus 4.6)

openshift-ci-robot · 2026-06-30T19:06:58Z

@sg00dwin: This pull request references Jira Issue OCPBUGS-38676, which is invalid:

expected the bug to target either version "4.22." or "openshift-4.22.", but it targets "5.0.0" instead
expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-38676 to depend on a bug targeting a version in 5.0.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Cherry-pick of #1164 to release-4.22.

Automated cherry-pick failed due to a merge conflict in pkg/console/operator/sync_v400.go (the ApplyConfigMap call site). Resolved manually — took the incoming retry wrapper.

Original PR

#1164

Test plan

Build clean

All unit tests pass

Conflict resolved: single site in sync_v400.go where ApplyConfigMap needed retry wrapper

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-30T19:07:00Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b12d131c-22a4-4170-a4cb-6afa72ff512a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci-robot · 2026-06-30T19:08:30Z

@sg00dwin: This pull request references Jira Issue OCPBUGS-94186, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Manual cherry-pick of #1164 to release-4.22.

Automated cherry-pick failed due to a merge conflict in pkg/console/operator/sync_v400.go (the ApplyConfigMap call site). Resolved manually — took the incoming retry wrapper.

Original PR

#1164

Test plan

Build clean

All unit tests pass

Conflict resolved: single site in sync_v400.go where ApplyConfigMap needed retry wrapper

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-06-30T19:24:25Z

@sg00dwin: This pull request references Jira Issue OCPBUGS-38676, which is invalid:

expected the bug to target either version "4.22." or "openshift-4.22.", but it targets "5.0.0" instead
expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead
release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-38676 to depend on a bug targeting a version in 5.0.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Summary

Manual cherry-pick of #1164 to release-4.22.

Automated cherry-pick failed due to a merge conflict in pkg/console/operator/sync_v400.go (the ApplyConfigMap call site). Resolved manually — took the incoming retry wrapper.

Original PR

#1164

Test plan

Build clean

All unit tests pass

Conflict resolved: single site in sync_v400.go where ApplyConfigMap needed retry wrapper

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sg00dwin · 2026-07-01T13:20:06Z

/test e2e-aws-console

sg00dwin · 2026-07-01T14:46:15Z

/jira refresh

openshift-ci-robot · 2026-07-01T14:46:19Z

@sg00dwin: This pull request references Jira Issue OCPBUGS-94186, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sg00dwin · 2026-07-01T14:54:34Z

/jira refresh

openshift-ci-robot · 2026-07-01T14:54:44Z

@sg00dwin: This pull request references Jira Issue OCPBUGS-94186, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-38676 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-38676 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
bug has dependents

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sg00dwin · 2026-07-01T18:00:12Z

/test e2e-aws-console

sg00dwin · 2026-07-01T21:02:28Z

/test e2e-aws-console

sg00dwin · 2026-07-02T03:17:59Z

/test e2e-aws-console

jhadvig

/lgtm
/approve

jhadvig · 2026-07-02T12:58:45Z

/label backport-risk-assessed
/verified by CI

openshift-ci · 2026-07-02T12:58:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig, sg00dwin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2026-07-02T12:58:59Z

@jhadvig: This PR has been marked as verified by CI.

Details

In response to this:

/label backport-risk-assessed
/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

sg00dwin · 2026-07-02T13:11:26Z

/test e2e-aws-console

sg00dwin · 2026-07-02T16:57:17Z

/test e2e-aws-console

openshift-merge-bot · 2026-07-02T20:22:41Z

/retest-required

Remaining retests: 0 against base HEAD 59eb087 and 2 for PR HEAD 9df61a7 in total

openshift-ci · 2026-07-02T23:08:08Z

@sg00dwin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-console	`9df61a7`	link	true	`/test e2e-aws-console`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

sg00dwin added 3 commits June 30, 2026 15:04

OCPBUGS-38676: Address review feedback on retry logic

9df61a7

- Guard deleteStorageVersionMigration against NotFound errors - Remove one-line retryOnTransientError wrapper, call util directly - Remove misleading nil-error test case from IsRetryableError Assisted-by: Claude (Opus 4.6)

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 30, 2026

openshift-ci Bot requested review from TheRealJon and jhadvig June 30, 2026 19:07

sg00dwin changed the title ~~OCPBUGS-38676: Add retry for transient API errors to prevent Degraded blips~~ OCPBUGS-94186: Add retry for transient API errors to prevent Degraded blips Jun 30, 2026

sg00dwin changed the title ~~OCPBUGS-94186: Add retry for transient API errors to prevent Degraded blips~~ [release-4.22] OCPBUGS-94186: Add retry for transient API errors to prevent Degraded blips Jul 1, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 1, 2026

jhadvig approved these changes Jul 2, 2026

View reviewed changes

openshift-ci Bot assigned jhadvig Jul 2, 2026

openshift-ci Bot added backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. lgtm Indicates that a PR is ready to be merged. labels Jul 2, 2026

openshift-ci Bot assigned sanketpathak Jul 2, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 2, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 2, 2026

Uh oh!

Conversation

sg00dwin commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Original PR

Test plan

Uh oh!

openshift-ci-robot commented Jun 30, 2026

Summary

Original PR

Test plan

Uh oh!

coderabbitai Bot commented Jun 30, 2026

Review skipped

Uh oh!

openshift-ci-robot commented Jun 30, 2026

Summary

Original PR

Test plan

Uh oh!

openshift-ci-robot commented Jun 30, 2026

Summary

Original PR

Test plan

Uh oh!

sg00dwin commented Jul 1, 2026

Uh oh!

sg00dwin commented Jul 1, 2026

Uh oh!

openshift-ci-robot commented Jul 1, 2026

Uh oh!

sg00dwin commented Jul 1, 2026

Uh oh!

openshift-ci-robot commented Jul 1, 2026

Uh oh!

sg00dwin commented Jul 1, 2026

Uh oh!

sg00dwin commented Jul 1, 2026

Uh oh!

sg00dwin commented Jul 2, 2026

Uh oh!

jhadvig left a comment

Choose a reason for hiding this comment

Uh oh!

jhadvig commented Jul 2, 2026

Uh oh!

openshift-ci Bot commented Jul 2, 2026

Uh oh!

openshift-ci-robot commented Jul 2, 2026

Uh oh!

sg00dwin commented Jul 2, 2026

Uh oh!

sg00dwin commented Jul 2, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 2, 2026

Uh oh!

openshift-ci Bot commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sg00dwin commented Jun 30, 2026 •

edited

Loading