After installing custom certificates in VCF Automation 9.1, VKS cluster status appears as "Unknown" on Kubernetes Management UI

Products

• VCF Automation
• VMware vSphere Kubernetes Service

Issue/Introduction

If the issuer of the VCFA serving certificate is changed, the cluster status will be displayed as "Unknown" on the "Kubernetes Management" UI.

vks-agent on the VKS cluster will report the following error.

kubectl logs -n vmware-system-vksm -l app=vks-agent
...
{"error":"x509: certificate signed by unknown authority","level":"error","msg":"verify TLS connection","sub-component":"tmc-context","time":"YYYY-MM-DDThh:mm:ssZ"}
{"component":"cluster-health","error":"could not connect to VKSM endpoint: context deadline exceeded","level":"error","msg":"init connection","sub-component":"event-stream","time":"YYYY-MM-DDThh:mm:ssZ"}
{"component":"cluster-health","error":"could not connect to VKSM endpoint: context deadline exceeded","level":"error","msg":"reconnect","sub-component":"event-stream","time":"YYYY-MM-DDThh:mm:ssZ"}

Running the following command on VCF Automation confirms that the new certificates are recognized.

kubectl get secret -n prelude ingress-vcfa-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl storeutl -certs -noout -text /dev/stdin | egrep 'Subject:|Issuer:'
# The output will display the new issuer.

However, running the following command on the Supervisor shows that the old certificates still remains.

NS=$(kubectl get ns -o custom-columns=:metadata.name | grep auto-attach)
kubectl get -n $NS secret auto-attach-secret -o jsonpath='{.datopenssl storeutl -certs -noout -text /dev/stdin | egrep 'Subject:|Issuer:'
# The output will display the old issuer, which is typically the management vCenter.

Environment

VCF Automation 9.1

Cause

The trust chain of the new certificate has not propagated to auto-attach service on the Supervisor.

Resolution

Reinstall auto-attach service.

This workaround will not impact your workload.
However, if you have made any manual modifications to the auto-attach service (such as the workaround described in KB 412306), re-apply those changes after implementing this workaround.

1. SSH into VCF Automation.
   *Log in as vmware-system-user.
   
   
2. Run sudo -i.
   *Enter the password for vmware-system-user when prompted.
   
   
3. Enter and run the following four lines:
   

   for name in $(kubectl get supervisorregistries.vcd.addon.vcf.vmware.com -n prelude -o name); do
     kubectl annotate -n prelude "$name" \
       reconcile-trigger="$(date +%s)" --overwrite
   done

   *The expected output is as follows:
   supervisorregistry.vcd.addon.vcf.vmware.com/vcfa-service-manager-registry-* annotated

4. Login to the VCF Automation Provider UI.
5. Navigate to [Service Management].
6. Go to "VKS cluster management" -> "ACTIONS" and execute "UNINSTALL".
   *This corresponds internally to the auto-attach service.
   *Please wait for the uninstallation to complete.
7. Go to "VKS cluster management" -> "ACTIONS" and execute "INSTALL".
   *You can keep the default values for all settings.
   *Please wait for the installation to complete.
8. Verify that the issue is resolved by checking the VCF Automation UI and the vks-agent logs.

Additional Information

For VCF Automation 9.0, refer Update the Certificate Trust Chain for VCFA Serving Certificates