TKG cluster upgrade stuck due to missing AntreaConfig

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• A Tanzu Kubernetes Grid (TKG) cluster upgrade remains in a stuck state.
• Describing the impacted cluster using kubectl describe cluster <cluster> reveals the following message:

  message: 'hook "BeforeClusterUpgrade" is blocking: annotation [before-upgrade.hook.cluster.cluster.x-k8s.io/vks-addon-pkgi] is set'

• The tanzu-addons-controller reports below error:

  # kubectl logs -n svc-tkg-domain-c## tanzu-addons-controller-manager-###########-#### | grep -i <guest-cluster-name>
  ...
  E0710 ##:##:##.######       1 clusterbootstrap_controller.go:601] "failed to update clusterBootstrap spec" err="failed to patch ClusterBootstrap <namespace>/<guest-cluster-name>: admission webhook \"clusterbootstrap.validating.vmware.com\" denied the request: ClusterBootstrap.run.tanzu.vmware.com \"<guest-cluster-name>\" is invalid: spec.cni.valuesFrom.ProviderRef: Invalid value: v1.TypedLocalObjectReference{APIGroup:(*string)(0xc0036f6ab0), Kind:\"AntreaConfig\", Name:\"<guest-cluster-name>-antrea-package\"}: antreaconfigs.cni.tanzu.vmware.com \"<guest-cluster-name>-antrea-package\" not found" logger="ClusterBootstrapController" cluster-ns="<namespace>" cluster-name="<guest-cluster-name>"

Environment

VMware vSphere Kubernetes Service

Cause

The exact cause has not been identified, but there might be two possible causes:

1. Interrupted network connectivity during the cluster upgrade between NCP pods on the Guest Clusters and the NSX Manager has been seen to contribute to this problem.
2. The AntreaConfig might have been accidentally deleted from within the guest cluster through other means.

Resolution

When experiencing this issue, take the following steps:

1. Check the firewall
   • Ensure firewall configurations allow traffic from Guest Cluster nodes to the NSX Manager on ports TCP/443, TCP/1234, and TCP/1235. Review latest requirements described in VMware Ports and Protocols (search for "Antrea" in the respective columns).
   • Verify connectivity using:

     # nc -vz <NSX-MANAGER-IP> 443

2. Manually re-create the AntreaConfig:
   • If the AntreaConfig is missing, re-create it by applying a manifest based on a healthy node or a standard template.
   • Use the following command:

     # kubectl apply -f antreaconfig.yaml --server-side

   • Ensure the naming convention matches the cluster name (e.g., <cluster-name>-antrea-package).
3. Verify the upgrade progress:
   • Monitor the tanzu-addons-controller logs to ensure reconciliation resumes.
   • Confirm worker nodes begin updating and Antrea packages are successfully deployed.

For further assistance, contact Broadcom support.