Skip to content

OCPBUGS-60157: ensure that images are rebuilt if they do not exist #5268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

OCPBUGS-60157: ensure that images are rebuilt if they do not exist #5268

wants to merge 5 commits into from

Conversation

cheesesashimi
Copy link
Member

@cheesesashimi cheesesashimi commented Aug 26, 2025 •
edited
Loading

- What I did

  • Improves image registry error handling by establishing additional error checking functions to determine whether the cause of an error is due to the image not existing or access being denied. Additionally, the build controller reconciler logic has been reworked to better detect and handle cases where an image is not found.
  • The OCL e2e test suite has been fixed by having the getPodFromJob() function check the init container status as opposed to the overall pod status. Additional improvements include consolidating the kernel type test with the rollout test which saves ~20 minutes of test execution time.
  • The ImagePruner e2e test suite has been augmented with more test cases as well as the addition of a test which targets the internal image registry where possible. This required a bit of work to ensure that skopeo is installed since the test has a dependency upon that.

- How to verify it

  1. Bring up a cluster.
  2. Opt the cluster into OCL.
  3. Run the TestMissingImageIsRebuilt as well as the TestImagePrunerOnCluster e2e tests against it.

- Description for the changelog
Ensure that missing images are rebuilt

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 26, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 26, 2025
@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 26, 2025
@openshift-ci-robot
Copy link
Contributor

@cheesesashimi: This pull request references Jira Issue OCPBUGS-60157, which is invalid:

  • expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

- What I did

TBD

- How to verify it

TBD

- Description for the changelog

TBD

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 26, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 26, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheesesashimi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2025
@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch 2 times, most recently from 66ffb33 to dafe85b Compare August 26, 2025 20:37
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 26, 2025
@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from dafe85b to 6682fc1 Compare August 26, 2025 21:16
@cheesesashimi
Copy link
Member Author

/test unit
/test verify
/test e2e-gcp-op-ocl

@cheesesashimi
detect if image exists or access is denied
ab216a9 
This also adds e2e tests for edge-cases that may have been missed as
well as a test-case that exposes the clusters' internal image registry
(if it exists) and validates the error handling functions against it.

The exposure functions were repackaged from the devex helpers for easy
reuse across both contexts.
@cheesesashimi
check whether the image exists before trying to reuse it for MachineC…
af4afce 
…onfigPools
@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from 6682fc1 to af4afce Compare August 26, 2025 22:00
@cheesesashimi
Copy link
Member Author

/test unit
/test verify
/test e2e-gcp-op-ocl

@isabella-janssen
Copy link
Member

/payload-job periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-aws-mco-disruptive

Running a payload test for this change to see if it impacts the regression tests positively.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 28, 2025

@isabella-janssen: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-4.20-periodics-e2e-aws-mco-disruptive

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c8d187c0-8415-11f0-8c7c-cabcfa25978b-0

@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl unit

@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl unit

@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl unit

@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from ebd9790 to 6b48b4e Compare September 9, 2025 19:41
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from 6b48b4e to 13c69c8 Compare September 9, 2025 19:52
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@cheesesashimi
Copy link
Member Author

/test unit

@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from 13c69c8 to b184612 Compare September 10, 2025 14:11
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@yuqi-zhang
Copy link
Contributor

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 10, 2025
@openshift-ci-robot
Copy link
Contributor

@yuqi-zhang: This pull request references Jira Issue OCPBUGS-60157, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from sergiordlr September 10, 2025 15:41
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from b44bec0 to d7da39e Compare September 11, 2025 17:18
@cheesesashimi
Copy link
Member Author

/test verify

@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from 793a491 to d7da39e Compare September 11, 2025 17:41
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

1 similar comment
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@cheesesashimi cheesesashimi force-pushed the zzlotnik/fix-image-registry-error-handling branch from 9a60c19 to 034b109 Compare September 12, 2025 18:12
@cheesesashimi
Copy link
Member Author

/test e2e-gcp-op-ocl

@openshift-ci-robot
Copy link
Contributor

@cheesesashimi: This pull request references Jira Issue OCPBUGS-60157, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

- What I did

  • Improves image registry error handling by establishing additional error checking functions to determine whether the cause of an error is due to the image not existing or access being denied. Additionally, the build controller reconciler logic has been reworked to better detect and handle cases where an image is not found.
  • The OCL e2e test suite has been fixed by having the getPodFromJob() function check the init container status as opposed to the overall pod status. Additional improvements include consolidating the kernel type test with the rollout test which saves ~20 minutes of test execution time.
  • The ImagePruner e2e test suite has been augmented with more test cases as well as the addition of a test which targets the internal image registry where possible. This required a bit of work to ensure that skopeo is installed since the test has a dependency upon that.

- How to verify it

  1. Bring up a cluster.
  2. Opt the cluster into OCL.
  3. Run the TestMissingImageIsRebuilt as well as the TestImagePrunerOnCluster e2e tests against it.

- Description for the changelog
Ensure that missing images are rebuilt

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@cheesesashimi cheesesashimi marked this pull request as ready for review September 12, 2025 18:23
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 12, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 12, 2025

@cheesesashimi: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/bootstrap-unit 034b109 link false /test bootstrap-unit
ci/prow/e2e-gcp-mco-disruptive 034b109 link false /test e2e-gcp-mco-disruptive
ci/prow/e2e-hypershift 034b109 link true /test e2e-hypershift
ci/prow/e2e-gcp-op-ocl 034b109 link false /test e2e-gcp-op-ocl
ci/prow/e2e-aws-ovn-upgrade-out-of-change 034b109 link false /test e2e-aws-ovn-upgrade-out-of-change
ci/prow/e2e-aws-mco-disruptive 034b109 link false /test e2e-aws-mco-disruptive
ci/prow/e2e-gcp-op-single-node 034b109 link true /test e2e-gcp-op-single-node
ci/prow/e2e-azure-ovn-upgrade-out-of-change 034b109 link false /test e2e-azure-ovn-upgrade-out-of-change
ci/prow/e2e-aws-ovn-upgrade 034b109 link true /test e2e-aws-ovn-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sergiordlr sergiordlr Awaiting requested review from sergiordlr

@djoshy djoshy Awaiting requested review from djoshy

@isabella-janssen isabella-janssen Awaiting requested review from isabella-janssen

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@cheesesashimi @openshift-ci-robot @isabella-janssen @yuqi-zhang @openshift-merge-robot