A lot of error message in the addon manager when there is new template type addon enabled

[zhujian7]

Describe the bug

After the ocm is installed, install the managed serviceaccount addon with template mode:

helm install \
            -n open-cluster-management-addon --create-namespace \
            managed-serviceaccount charts/managed-serviceaccount/ \
            --set tag=latest \
            --set featureGates.ephemeralIdentity=true \
            --set hubDeployMode=AddOnTemplate \
            --set targetCluster=cluster1

Check the logs of the cluster-manager-addon-manager:

I0911 03:40:14.794432       1 controller.go:175] "Starting an addon manager for addon" addonName="managed-serviceaccount"
I0911 03:40:14.794526       1 template_agent.go:283] "Addon template config in addon status is empty" addonName="managed-serviceaccount"
E0911 03:40:14.794547       1 template_agent.go:160] "Unhandled Error" err="GetAgentAddonOptions addon managed-serviceaccount template is nil"
I0911 03:40:14.799079       1 template_agent.go:283] "Addon template config in addon status is empty" addonName="managed-serviceaccount"
E0911 03:40:14.799097       1 template_agent.go:160] "Unhandled Error" err="GetAgentAddonOptions addon managed-serviceaccount template is nil"
I0911 03:40:14.800326       1 template_agent.go:283] "Addon template config in addon status is empty" addonName="managed-serviceaccount"
E0911 03:40:14.800359       1 template_agent.go:160] "Unhandled Error" err="GetAgentAddonOptions addon managed-serviceaccount template is nil"
I0911 03:40:14.800975       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:14.801029       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: placement.cluster.open-cluster-management.io \"global\" not found"
I0911 03:40:14.801769       1 template_agent.go:289] "Addon template spec hash is empty" addonName="managed-serviceaccount"
E0911 03:40:14.801805       1 template_agent.go:156] "Unhandled Error" err="GetAgentAddonOptions failed to get addon managed-serviceaccount template: addon managed-serviceaccount template desired spec hash is empty"
I0911 03:40:14.803220       1 base_controller.go:34] Waiting for caches to sync for CSRSignController
I0911 03:40:14.803235       1 base_controller.go:34] Waiting for caches to sync for addon-deploy-controller
I0911 03:40:14.803240       1 base_controller.go:34] Waiting for caches to sync for CSRApprovingController
I0911 03:40:14.803248       1 base_controller.go:34] Waiting for caches to sync for management-addon-config-controller
I0911 03:40:14.803220       1 base_controller.go:34] Waiting for caches to sync for cma-managed-by-controller
I0911 03:40:14.803260       1 base_controller.go:40] Caches are synced for cma-managed-by-controller
I0911 03:40:14.803262       1 base_controller.go:34] Waiting for caches to sync for addon-config-controller
I0911 03:40:14.803265       1 base_controller.go:78] Starting #1 worker of cma-managed-by-controller controller ...
I0911 03:40:14.803250       1 base_controller.go:34] Waiting for caches to sync for addon-registration-controller
E0911 03:40:14.808562       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: placement.cluster.open-cluster-management.io \"global\" not found"
E0911 03:40:14.819946       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: placement.cluster.open-cluster-management.io \"global\" not found"
E0911 03:40:14.841309       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: placement.cluster.open-cluster-management.io \"global\" not found"
I0911 03:40:14.871330       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.880709       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:14.883924       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: Operation cannot be fulfilled on clustermanagementaddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0911 03:40:14.897340       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:14.897485       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.904039       1 base_controller.go:40] Caches are synced for management-addon-config-controller
I0911 03:40:14.904065       1 base_controller.go:78] Starting #1 worker of management-addon-config-controller controller ...
I0911 03:40:14.904070       1 base_controller.go:40] Caches are synced for addon-registration-controller
I0911 03:40:14.904081       1 base_controller.go:40] Caches are synced for CSRSignController
I0911 03:40:14.904092       1 base_controller.go:78] Starting #1 worker of addon-registration-controller controller ...
I0911 03:40:14.904096       1 base_controller.go:40] Caches are synced for addon-config-controller
I0911 03:40:14.904124       1 base_controller.go:40] Caches are synced for addon-deploy-controller
I0911 03:40:14.904151       1 base_controller.go:78] Starting #1 worker of addon-deploy-controller controller ...
I0911 03:40:14.904170       1 base_controller.go:78] Starting #1 worker of addon-config-controller controller ...
I0911 03:40:14.904099       1 base_controller.go:78] Starting #1 worker of CSRSignController controller ...
I0911 03:40:14.904112       1 base_controller.go:40] Caches are synced for CSRApprovingController
I0911 03:40:14.904195       1 base_controller.go:78] Starting #1 worker of CSRApprovingController controller ...
I0911 03:40:14.904193       1 template_agent.go:289] "Addon template spec hash is empty" addonName="managed-serviceaccount"
E0911 03:40:14.904223       1 template_agent.go:156] "Unhandled Error" err="GetAgentAddonOptions failed to get addon managed-serviceaccount template: addon managed-serviceaccount template desired spec hash is empty"
I0911 03:40:14.904967       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.909446       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:14.910000       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:14.911205       1 template_agent.go:289] "Addon template spec hash is empty" addonName="managed-serviceaccount"
E0911 03:40:14.911244       1 template_agent.go:156] "Unhandled Error" err="GetAgentAddonOptions failed to get addon managed-serviceaccount template: addon managed-serviceaccount template desired spec hash is empty"
I0911 03:40:14.914777       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.920048       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:14.922172       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0911 03:40:14.923228       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:14.924312       1 template_agent.go:289] "Addon template spec hash is empty" addonName="managed-serviceaccount"
I0911 03:40:14.929412       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:14.939307       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0911 03:40:14.939962       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:14.945734       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.957098       1 registration.go:466] "Rolebinding for addon updated" namespace="cluster1" name="open-cluster-management:managed-serviceaccount:clusterrole:agent" clusterName="cluster1" addonName="managed-serviceaccount"
I0911 03:40:14.957202       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"open-cluster-management-hub", Name:"cluster-manager-addon-manager-controller", UID:"d0ac0339-a8a3-40e1-af97-d5d449de4c32", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RoleBindingCreated' Created RoleBinding.rbac.authorization.k8s.io/open-cluster-management:managed-serviceaccount:clusterrole:agent -n cluster1 because it was missing
I0911 03:40:14.959221       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:14.970823       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.975562       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
E0911 03:40:14.979578       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:14.991198       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:14.999916       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:15.000391       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0911 03:40:15.004738       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:15.009858       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:15.010590       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.012185       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:15.014032       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:15.077727       1 csr_helpers.go:174] CSR approved
I0911 03:40:15.102233       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.113533       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.126241       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:15.126423       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:15.136274       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:15.139889       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.160447       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
E0911 03:40:15.161024       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:15.161909       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.163363       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:15.190670       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
E0911 03:40:15.194565       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:15.194837       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
I0911 03:40:15.197933       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.204191       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.205657       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
E0911 03:40:15.209180       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0911 03:40:15.218212       1 controller.go:171] "There already is a manager started for addon, skipping" addonName="managed-serviceaccount"
I0911 03:40:15.219994       1 addon_config.go:56] "Addon deployment config is nil, return an empty string for agent install namespace" addonNamespace="cluster1" addonName="managed-serviceaccount"
E0911 03:40:15.222932       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

No template desired spec hash is empty and template is nil errors.

Environment ie: OCM version, Kubernetes version and provider:

Additional context
Add any other context about the problem here.

[dosubot]

The errors you're seeing—such as "Addon template config in addon status is empty", "template is nil", "template desired spec hash is empty", and "placement.cluster.open-cluster-management.io \"global\" not found"—are all related to missing or incomplete configuration for template type addons in OCM.

Here's what's happening and how to address it:

1. Missing AddOnTemplate config or spec hash:
   These errors mean the ManagedClusterAddOn status is missing a reference to the AddOnTemplate or its spec hash is empty. This usually happens if the AddOnTemplate resource isn't created, isn't referenced correctly in the ClusterManagementAddOn, or the creation order is wrong. Make sure the ClusterManagementAddOn is created first, then the AddOnTemplate, and that supportedConfigs in ClusterManagementAddOn references the AddOnTemplate by name. See issue #993 and template_agent.go.

2. Placement resource not found:
   The error about "placement.cluster.open-cluster-management.io \"global\" not found" means the Placement resource named global is missing. If your ClusterManagementAddOn uses an installStrategy with placements, you must create the referenced Placement resource in the correct namespace. Without it, the addon manager can't select clusters for deployment. See placement docs.

3. Resource version conflicts:
   Errors like "Operation cannot be fulfilled ... the object has been modified; please apply your changes to the latest version and try again" are common in Kubernetes when multiple controllers update the same resource. These are usually transient but can be exacerbated by repeated reconcile errors.

4. Recent bug and fix:
   There was a bug where the template agent's CSR functions would fall back to the default ClusterManagementAddOn template instead of using the specific ManagedClusterAddOn template, causing template config errors (issue #1179). Also, a recent fix (PR #1160) improved the addon template controller to avoid stopping addon managers prematurely, which helps reduce duplicate manager and resource conflict errors.

Troubleshooting steps:

• Ensure the ClusterManagementAddOn and AddOnTemplate resources are created in the correct order and reference each other properly.
• Verify that the Placement resource referenced in your ClusterManagementAddOn exists and is correctly configured.
• Check that the ManagedClusterAddOn status includes a valid configReference to the AddOnTemplate and that the spec hash is populated.
• If you recently upgraded OCM or the addon-framework, make sure you have the latest bug fixes.

If you address the missing or misconfigured resources, these errors should resolve and the addon manager should deploy the managed-serviceaccount addon as expected. For more details, see the addon template documentation and placement API docs.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[zhujian7]

I think the root cause is the controllers(in the addon framework) that the addon-manager started, processing the template-type managedclusteraddon too early. It should wait until the managedclusteraddon.status.configReferences has the proper addontemplates set, then start to handle their logic.

[zhujian7]

After the fix, there is no such error "Addon template config in addon status is empty", "template is nil", "template desired spec hash is empty", only some "the object has been modified; please apply your changes to the latest version and try again" left

I0916 06:51:33.584026       1 base_controller.go:76] Waiting for caches to sync for addon-management-controller
I0916 06:51:33.584040       1 base_controller.go:76] Waiting for caches to sync for addon-configuration-controller
I0916 06:51:33.584041       1 base_controller.go:76] Waiting for caches to sync for addon-progressing-controller
I0916 06:51:33.584029       1 base_controller.go:76] Waiting for caches to sync for addon-owner-controller
I0916 06:51:33.584069       1 base_controller.go:76] Waiting for caches to sync for addon-template-controller
I0916 06:51:33.584075       1 base_controller.go:76] Waiting for caches to sync for cma-install-progression-controller
I0916 06:51:33.684485       1 base_controller.go:82] Caches are synced for addon-template-controller
I0916 06:51:33.684502       1 base_controller.go:119] Starting #1 worker of addon-template-controller controller ...
I0916 06:51:33.684510       1 base_controller.go:82] Caches are synced for addon-management-controller
I0916 06:51:33.684522       1 base_controller.go:119] Starting #1 worker of addon-management-controller controller ...
I0916 06:51:33.684528       1 base_controller.go:119] Starting #2 worker of addon-management-controller controller ...
I0916 06:51:33.684554       1 controller.go:175] "Starting an addon manager for addon" addonName="managed-serviceaccount"
I0916 06:51:33.684574       1 base_controller.go:82] Caches are synced for addon-configuration-controller
I0916 06:51:33.684579       1 base_controller.go:119] Starting #1 worker of addon-configuration-controller controller ...
I0916 06:51:33.684588       1 base_controller.go:119] Starting #2 worker of addon-configuration-controller controller ...
I0916 06:51:33.684645       1 base_controller.go:82] Caches are synced for addon-owner-controller
I0916 06:51:33.684662       1 base_controller.go:82] Caches are synced for addon-progressing-controller
I0916 06:51:33.684671       1 base_controller.go:119] Starting #1 worker of addon-progressing-controller controller ...
I0916 06:51:33.684664       1 base_controller.go:119] Starting #1 worker of addon-owner-controller controller ...
I0916 06:51:33.684675       1 base_controller.go:119] Starting #2 worker of addon-progressing-controller controller ...
I0916 06:51:33.684684       1 base_controller.go:119] Starting #2 worker of addon-owner-controller controller ...
I0916 06:51:33.684720       1 base_controller.go:82] Caches are synced for cma-install-progression-controller
I0916 06:51:33.684770       1 base_controller.go:119] Starting #1 worker of cma-install-progression-controller controller ...
I0916 06:51:33.684779       1 base_controller.go:119] Starting #2 worker of cma-install-progression-controller controller ...
I0916 06:51:33.688870       1 base_controller.go:34] Waiting for caches to sync for addon-config-controller
I0916 06:51:33.688895       1 base_controller.go:34] Waiting for caches to sync for addon-deploy-controller
I0916 06:51:33.688920       1 base_controller.go:34] Waiting for caches to sync for CSRSignController
I0916 06:51:33.688942       1 base_controller.go:34] Waiting for caches to sync for CSRApprovingController
I0916 06:51:33.689089       1 base_controller.go:34] Waiting for caches to sync for management-addon-config-controller
I0916 06:51:33.689093       1 base_controller.go:34] Waiting for caches to sync for addon-registration-controller
I0916 06:51:33.689122       1 base_controller.go:34] Waiting for caches to sync for cma-managed-by-controller
I0916 06:51:33.689224       1 base_controller.go:40] Caches are synced for cma-managed-by-controller
I0916 06:51:33.689232       1 base_controller.go:78] Starting #1 worker of cma-managed-by-controller controller ...
I0916 06:51:33.789439       1 base_controller.go:40] Caches are synced for addon-deploy-controller
I0916 06:51:33.789454       1 base_controller.go:78] Starting #1 worker of addon-deploy-controller controller ...
I0916 06:51:33.789470       1 base_controller.go:40] Caches are synced for addon-config-controller
I0916 06:51:33.789473       1 base_controller.go:78] Starting #1 worker of addon-config-controller controller ...
I0916 06:51:33.789539       1 base_controller.go:40] Caches are synced for CSRApprovingController
I0916 06:51:33.789560       1 base_controller.go:78] Starting #1 worker of CSRApprovingController controller ...
I0916 06:51:33.789581       1 base_controller.go:40] Caches are synced for CSRSignController
I0916 06:51:33.789591       1 base_controller.go:78] Starting #1 worker of CSRSignController controller ...
I0916 06:51:33.789610       1 base_controller.go:40] Caches are synced for addon-registration-controller
I0916 06:51:33.789628       1 base_controller.go:78] Starting #1 worker of addon-registration-controller controller ...
I0916 06:51:33.789646       1 base_controller.go:40] Caches are synced for management-addon-config-controller
I0916 06:51:33.789674       1 base_controller.go:78] Starting #1 worker of management-addon-config-controller controller ...
E0916 06:53:20.245734       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0916 06:53:20.246006       1 base_controller.go:277] "Unhandled Error" err="\"addon-progressing-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0916 06:53:20.254207       1 base_controller.go:277] "Unhandled Error" err="\"addon-configuration-controller\" controller failed to sync \"managed-serviceaccount\", err: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0916 06:58:20.081991       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: failed to patch status(supported configs) of managedclusteraddon: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0916 06:58:20.094500       1 registration.go:477] "Rolebinding for addon updated" namespace="cluster1" name="open-cluster-management:managed-serviceaccount:clusterrole:agent" clusterName="cluster1" addonName="managed-serviceaccount"
I0916 06:58:20.094573       1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"open-cluster-management-hub", Name:"cluster-manager-addon-manager-controller", UID:"d0ac0339-a8a3-40e1-af97-d5d449de4c32", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RoleBindingCreated' Created RoleBinding.rbac.authorization.k8s.io/open-cluster-management:managed-serviceaccount:clusterrole:agent -n cluster1 because it was missing
E0916 06:58:20.109634       1 base_controller.go:154] "Unhandled Error" err="\"addon-registration-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: failed to patch status condition(successfully configured) of managedclusteraddon: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0916 06:58:20.126693       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: failed to update addon cluster1/managed-serviceaccount: failed to update addon status: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
I0916 06:58:20.215675       1 csr_helpers.go:180] CSR approved
E0916 06:58:20.315009       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: failed to update addon cluster1/managed-serviceaccount: failed to update addon status: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"
E0916 06:58:20.331911       1 base_controller.go:154] "Unhandled Error" err="\"addon-deploy-controller\" controller failed to sync \"cluster1/managed-serviceaccount\", err: failed to update addon cluster1/managed-serviceaccount: failed to update addon status: Operation cannot be fulfilled on managedclusteraddons.addon.open-cluster-management.io \"managed-serviceaccount\": the object has been modified; please apply your changes to the latest version and try again"