Error: "ServiceUnavailable" and "Bad Status 404" on Antrea when upgrading a vSphere Supervisor workload cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During a vSphere Supervisor workload cluster upgrade, cluster conditions report unhealthy status for Antrea. The vSphere Kubernetes Cluster reflects a Ready False state. While the workload cluster successfully deploys nodes and pods, the kube-apiserver fails to process requests for Antrea CustomResourceDefinitions (CRDs), resulting in a "ServiceUnavailable" error or intermittent Bad Status 404 responses. The issue manifests with the following symptoms:

• From the Supervisor cluster context, describing the cluster returns either of the below unhealthy conditions:
  
  kubectl describe cluster <cluster_name> -n <namespace>
  

  message: ClusterBootstrap conditions Antrea-ReconcileFailed indicate reconcile has failed
    reason: ClusterBootstrapFailed

  reason: ContainerNetworkingNotInstalled
    severity: Warning
    status: "False"
    type: NetworkProviderReconciled

• From the Supervisor cluster context, describing the clusterbootstrap for the affected workload cluster returns the similar kapp errors to the below:
  
  kubectl describe clusterbootstrap <clusterbootstrap_name> -n <namespace>

message: |-
      kapp: Error: Timed out waiting after 30s for resources:
        - apiservice/v1beta1.system.antrea.io (apiregistration.k8s.io/v1) cluster
        - apiservice/v1beta2.controlplane.antrea.io (apiregistration.k8s.io/v1) cluster
        - apiservice/v1alpha1.stats.antrea.io (apiregistration.k8s.io/v1) cluster
status: "True"
    type: Antrea-ReconcileFailed

• Within the affected workload cluster's context, trying any kubectl commands on the antrea CustomResourceDefintions (CRDs) associated with the above affected antrea APIservices returns an error message similar to the below:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

• Within the affected workload cluster's context, performing a describe on the antrea app shows that one or more antrea APIservices Available False state with a FailedDiscoveryCheck error.

kubectl get apiservice | grep antrea

v1alpha1.crd.antrea.io                   Local                        True
v1alpha1.crd.antrea.tanzu.vmware.com     Local                        True
v1alpha1.stats.antrea.io                 kube-system/antrea           False (FailedDiscoveryCheck)
v1alpha2.crd.antrea.io                   Local                        True
v1alpha3.crd.antrea.io                   Local                        True
v1beta1.crd.antrea.io                    Local                        True
v1beta1.system.antrea.io                 kube-system/antrea           False (FailedDiscoveryCheck)
v1beta2.controlplane.antrea.io           kube-system/antrea           False (FailedDiscoveryCheck)

kubectl describe app -n vmware-system-tkg <antrea app>

 conditions:
  - message: 'Deploying: Error (see .status.usefulErrorMessage for details)'
    status: "True"
    type: ReconcileFailed

 kapp: Error: Timed out waiting after 30s for resources:
        - apiservice/v1alpha1.stats.antrea.io (apiregistration.k8s.io/v1) cluster
        - apiservice/v1beta2.controlplane.antrea.io (apiregistration.k8s.io/v1) cluster
        - apiservice/v1beta1.system.antrea.io (apiregistration.k8s.io/v1) cluster

Namespace    Name                                Kind        Age  Op      Op st.  Wait to    Rs       Ri
      (cluster)    v1alpha1.stats.antrea.io            APIService  1y   -       -       reconcile  ongoing  Condition Available is not True
                                                                                                            (False)
      ^            v1beta1.system.antrea.io            APIService  1y   -       -       reconcile  ongoing  Condition Available is not True
                                                                                                            (False)
      ^            v1beta2.controlplane.antrea.io      APIService  1y   -       -       reconcile  ongoing  Condition Available is not True
                                                                                                            (False)

Environment

VMware vSphere Kubernetes Service

Cause

• When Antrea API resources are requested over the kube-apiserver, requests are routed to a service or virtual machine (VM) configured with an identical IP address to the Antrea service within the vSphere Supervisor workload cluster. This occurs because the kube-proxy system pod starts later than the kube-apiserver system pod on the node. Consequently, the established connection queries an incorrect service, which returns the noted error status messages. The erroneous connection persists because the service with the identical IP address is reachable and connections are not refused.
  
  
• The connection requires correction on all control plane nodes within the affected workload cluster. The control plane endpoint (VIP) load balances kube-apiserver requests across all control plane nodes. When the established connection is erroneous on a single node, requests fail intermittently upon being load-balanced to the affected control plane node with the faulty connection.

Resolution

The connection configured from the kube-apiserver to the Antrea service requires correction.

1. Connect into the affected workload cluster context.
   
   
2. Retrieve the list of all kube-apiserver pods in the workload cluster:
   
   kubectl get pods -n kube-system | grep "kube-apiserver"
   
   
3. Restart each kube-apiserver pod sequentially, allowing each pod to reach a running state before proceeding to the next:
   
   kubectl delete pod -n kube-system <kube-apiserver pod name>
   
   
4. Confirm the status of the Antrea daemonset, pkgi, and app, which shortly return True healthy and Reconcile Succeeded healthy states:
   
   kubectl get ds,pkgi,app -n kube-system | grep antrea
   
   
5. If the above steps do not resolve the issue, open a support request with Broadcom Technical Support and reference this article.

Additional Information

This issue has been flagged to upstream Kubernetes (Issue 135883)