VKR Upgrade cannot be initiated as cluster's AddonReconciled condition is not True
Article ID: 435160
Updated On:
Products
Issue/Introduction
Unable to start an upgrade for a vSphere Kubernetes Service (VKS) cluster because the system returns an error similar to one of the below:
Error: upgrade cannot be initiated as cluster's AddonsReconciled condition is not True.
Addon <addon name> is not ready: kapp: Error: Timed out waiting after 30s for resources: [deployment/<addon name> (apps/v1) namespace: <addon namespace>]
"admission webhook "capi.validating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: upgrade cannot be initiated as cluster's AddonsReconcile condition is not True. Message: Addon <addon name> is not ready: Reason: Reconciling"This error message can note replicaset instead of a deployment depending on the addon and progress:
Error: Timed out waiting after 30s for resources: [replicaset/<addon name with id> (apps/v1) namespace: <addon namespace>]Environment
vSphere Supervisor
vSphere Kubernetes Service (VKS)
Cause
System upgrade pre-checks have found that one or more addons within the affected VKS cluster are not healthy.
The specific error calls out the particular unhealthy addon and its namespace within the affected VKS cluster.
Addons correspond to installed packages (PKGI) that are manually installed or that are automatically installed and upgraded based on the vSphere Kubernetes Release (VKR) version.
- Manually installed addons are converted standard packages in vSphere Cloud Foundation 9 (VCF 9): VKS Standard Packages Release Notes
- Automatically installed addons and their versions are listed in the release notes for the respective VKR version: VMware VKR Release Notes
Resolution
The deployment in the error message must be investigated further to understand why it is failing.
- Connect into the affected VKS cluster's context
- Locate the deployment and corresponding replicaset(s) in the namespace from the error message:
kubectl get deployment,replicaset -n <addon namespace> - Describe the deployment and newest replicaset for more details:
kubectl describe deployment -n <addon namespace> <deployment name> kubectl describe replicaset -n <addon namespace> <replicaset name> - If the deployment shows 0 Ready, 0 Available but the replicaset shows X Current but 0 Ready, further checks can be made on the pod managed by the deployment and replicaset:
kubectl get pods -n <addon namespace> kubectl describe pod -n <addon namespace> <pod name> - See the below table for next steps based on the errors found from describing the deployment and replicaset accordingly.
| Error | Explanation and Recommendations |
| Warning FailedCreate - Error creating: pods "<addon name>" is forbidden: violates PodSecurity: <podSecurity setting> - pod or container <security pod> must set <SecurityContext requirement> | The noted <security pod> is preventing the addon's pod(s) from creating. Validate that the noted <security pod> is properly set to not block the creation of the addon. NOTE: VMware by Broadcom is not responsible for and does not provide support for third party applications. |
Pod in Pending state with Warning FailedScheduling 0 nodes are available
| A pod in Pending state is waiting for a node that matches its requirements. These requirements can include requested resources, tolerations and anti-affinity rules. The error message from describing the pod provides details on these requirements. Describing the pod will also provide details on the pod's configuration for these requirements. |
Additional Information
Starting in VKS 3.6, a --force option was added to allow VKR upgrades to progress when the addon's status is not Ready.
Note: This force option only applies to manually installed addons/standard packages. It will not apply to automatically installed addons which are system critical and packaged with each VKR version.
Fixed Issues under VKS 3.6 Release Notes: vSphere Kubernetes Service 3.6.0+v1.35
Japanese version: クラスターの AddonReconciled 条件が True ではないため、VKR アップグレードを開始できません。
Feedback
