vSphere Supervisor Workload Cluster Error: clusterclass is not successfully reconciled: status of VariablesReconciled condition on ClusterClass must be "True"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• All VKS Cluster operations (e.g., scaling, creating, upgrading) will fail to start

Example: Horizontally scale a VKS cluster worker nodes count by changing the number of nodes will not start. Nodes are not getting scaled up.

• The VKS cluster machine deployment (MD)  object will be "Running"  PHASE status and not "ScalingUp or ScalingDown" PHASE  and the number replicas didn't change.

# kubectl  get  md   <clusterName> -n <namespace>

NAME                                                             CLUSTER          REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE    VERSION
machinedeployment.cluster.x-k8s.io/clusterName-worker-l9crz      clusterName   5          5       5         0             Running   148d   v1.##.1+vmware.1-fips

• The describe of the cluster Kubernetes object for the VKS cluster will show Message "ClusterClass is not successfully reconciled: status of VariablesReconciled condition on ClusterClass must be "True"" or "ClusterClass is not successfully reconciled: status of VariablesReady condition on ClusterClass must be "True"

# kubectl describe cluster <clusterName> -n <namespace> 

      Message:
      Observed Generation:   12
      Reason:                Available
      Status:                True
      Type:                  WorkersAvailable
      Last Transition Time:  [timestamp]
      Message:               ClusterClass is not successfully reconciled: status of VariablesReconciled condition on ClusterClass must be "True"
      Observed Generation:   12
      Reason:                ReconcileFailed
      Status:                False
      Type:                  TopologyReconciled
      Last Transition Time:  [timestamp]
      Message:
      Observed Generation:   12
      Reason:                NotRollingOut
      Status:                False
      Type:                  RollingOut
      Last Transition Time:  [timestamp]

• Running the following command  show VariableDiscovery of the ClusterClass is failing since the connection to the runtime-extension-webhook-service.svc-tkg-domain-c## service is failing with error "unknown certificate authority"
  
  # kubectl get  cc -n svc-tkg-domain-c### builtin-generic-v3.3.0  -o jsonpath='{.status.conditions}' | jq
[
  {
    "lastTransitionTime": "[timestamp]",
    "status": "True",
    "type": "RefVersionsUpToDate"
  },
  {
    "lastTransitionTime": "[timestamp]",
    "message": "VariableDiscovery failed: failed to call DiscoverVariables for patch default: failed to call extension handler \"discover-variables.runtime-extension\": http call failed: Post \"https://runtime-extension-webhook-service.svc-tkg-domain-c##.svc:443/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovervariables/discover-variables?timeout=10s\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: invalid signature: parent certificate cannot sign this kind of certificate\" while trying to verify candidate authority certificate \"serial:340174157205######8522218632\")",
    "reason": "VariableDiscoveryFailed",
    "severity": "Error",
    "status": "False",
    "type": "VariablesReconciled"
  }
]
  

   

• The runtime-extension-controller-manager-########## pod logs show TLS error "failed to verify certificate: x509: certificate signed by unknown authority"
  
  # kubectl logs -n svc-tkg-domain-##  runtime-extension-controller-manager-###########

  [timestamp]       1 ???:1] "http: TLS handshake error from #.#.#.#:58377: tls: failed to verify certificate: x509: certificate signed by unknown authority"
[timestamp]       1 ???:1] "http: TLS handshake error from #.#.#.#:2134: tls: failed to verify certificate: x509: certificate signed by unknown authority"
[timestamp]       1 ???:1] "http: TLS handshake error from #.#.#.#:25239: tls: failed to verify certificate: x509: certificate signed by unknown authority"
  
  

• The capi-controller-manager pod logs showing connection to the runtime-extension-webhook-service.svc-tkg-domain-c## service IP is falling with TLS error "unknown certificate authority"

# kubectl logs -n  <svc-tkg-domain-c#>  capi-controller-manager########### -c manager

nHandler="discover-variables.runtime-extension" hook="DiscoverVariables"
[timestamp]       1 controller.go:347] "Reconciler error" err="failed to discover variables for ClusterClass builtin-generic-v3.1.0: failed to call DiscoverVariables for patch default: failed to call extension handler \"discover-variables.runtime-extension\": http call failed: Post \"https://runtime-extension-webhook-service.svc-tkg-domain-c##.svc:443/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovervariables/discover-variables?timeout=10s\": remote error: tls: unknown certificate authority" controller="clusterclass" controllerGroup="cluster.x-k8s.io" controllerKind="ClusterClass" ClusterClass="vmware-system-monitoring/builtin-generic-v3.1.0" namespace="vmware-system-monitoring" name="builtin-generic-v3.1.0" reconcileID="7f03f0f2-3bad-####-a7b8-a86e1edbf271"

Environment

VMware vSphere Kubernetes Service

VKS supervisor service 3.4.1 and higher

Cause

• Client certificate rotation is not handled correctly for the runtime-extension-controller system pod. This causes a mismatch between the certificate in the runtime-extension-webhook-service-cert secret and the certificate on the pod itself.
  
  
• The certificate in the secret will show newer 'notBefore' and 'notAfter' dates and a different serial number than the one assigned to the runtime-extension pod.
  
  
  • The runtime-extension-webhook-service-cert secret certificate:
    
    # kubectl get secret/runtime-extension-webhook-service-cert -n svc-tkg-domain-## -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates -serial

Ex:

# kubectl get secret/runtime-extension-webhook-service-cert -n svc-tkg-domain-c## -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates -serial

notBefore=Jan 31 07:01:27 #### GMT
notAfter=May  1 07:01:27 #### GMT
serial=14E01E25####5056BB1B0D86C271B96
    
    
  • The runtime-extension Pod certificate:
    
    # kubectl get node $(kubectl get pod <runtime-extension-controller-POD-Name> -n svc-tkg-domain-c8 -o jsonpath='{.spec.nodeName}') -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' | xargs -I {} sh -c "echo | openssl s_client -connect {}:9442 2>/dev/null | openssl x509 -noout -dates -serial"
    
    Ex:
    

    # kubectl get node $(kubectl get pod runtime-extension-controller-manager-6c###849-j2gww -n svc-tkg-domain-c8 -o jsonpath='{.spec.nodeName}') -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}' | xargs -I {} sh -c "echo | openssl s_client -connect {}:9442 2>/dev/null | openssl x509 -noout -dates -serial"

    notBefore=Jan  2 23:32:33 #### GMT
notAfter=Apr  2 23:32:33 #### GMT
serial=AB1B66F11F####78630996A9E4F

• Review the certificate by running the following command against the master node IP where the runtime-extension-controller pod is located (using port 9442):
          # echo | openssl s_client -connect <master-node-IP>:9442 2>/dev/null | openssl x509 -noout -dates -serial

Resolution

Resolution

This issue is resolved in vSphere Kubernetes Service 3.5.1+v1.34 and 3.6.0+v1.35.

Refer to VMware vSphere Kubernetes Service 3.5 Release Notes - Fixed Issues

 

Workaround

The system pod with the CA issue will need to be restarted to correct the certificate issue.

1. Connect into the Supervisor cluster context
   
   
2. Locate the svc-tkg domain namespace:
   

   kubectl get ns | grep "svc-tkg"
   
   svc-tkg-domain-c##        Active

3. Restart the runtime-extension-controller pod in the svc-tkg namespace:
   

   kubectl rollout restart deploy runtime-extension-controller-manager -n <svc-tkg-domain namespace>

   • If you do not have permissions for rollout restart, the pod can be deleted to restart it:
     

     kubectl get pods -n <svc-tkg-domain namespace> | grep runtime
     kubectl delete pod <runtime-extension-exact-pod-name> -n <svc-tkg-domain namespace>

     
     
     
4. Check that the runtime-extension-controller pod restarted successfully:
   

   kubectl get pods -n <svc-tkg-domain namespace> | grep runtime

 

For quicker reconciliation, the below steps can be performed:

1. Restart the capi-controller-manager pod:
   

   kubectl rollout restart deploy -n <svc-tkg-domain namespace> capi-controller-manager

   
   
   • If you do not have permissions for rollout restart, the pod can be deleted to restart it:
     

     kubectl get pods -n <svc-tkg-domain namespace> | grep capi-controller-manager
     kubectl delete pod <capi-controller-manager-exact-pod-name> -n <svc-tkg-domain namespace>

      

2. Check that the capi-controller-manager pods restarted successfully:
   

   kubectl get pods -n <svc-tkg-domain namespace> | grep capi-controller-manager

 

Validation Steps:

1. Confirm that the cluster no longer has the clusterClass error message post-restart of the above system pods:
   

   kubectl describe cluster <cluster name> -n <namespace>

Additional Information

• This certificate issue is expected to occur every 60 days, requiring a restart of the above pods.
  
  
• If the above steps do not correct the issue, reach out to VMware by Broadcom Technical Support and upload a Workload Management Supervisor Cluster Support Bundle. See Gathering Logs for vSphere with Tanzu