VKS Cluster Management Busy - auto-attach error "unexpected status code 502 Bad Gateway"

Products

VMware vSphere Kubernetes Service VCF Automation

Issue/Introduction

Unable to install VKS Cluster Management, it is stuck in "busy" state in the VMware Cloud Foundation Automation | Provider Management web UI.

Supervisor Management Proxy supervisor service is healthy and correctly configured as per the documentation: Configuring the Supervisor Management Proxy

When viewing details of the auto-attach service in Error state from the Supervisor cluster's supervisor services, the below error message is shown:

[ERROR] vendir: Error: Syncing directory '0':
        Syncing directory '.' with imgpkgBundle contents:
        Fetching image:
        Error while preparing a transport to talk with the registry:
        Unable to create round tripper:
        GET https://mgmt-image-proxy.kube-system.svc.cluster.local/v2/: unexpected status code 502 Bad Gateway
        <html>
        <head><title>502 Bad Gateway</title></head>
        <body>
        <center><h1>502 Bad Gateway</h1></center>
        <hr><center>nginx/1.##.2</center>
        </body>
        </html>       
        Get "http://mgmt-image-proxy.kube-system.svc.cluster.local/v2/": dial tcp <mgmt-image-proxy external IP>:80: connect: connection refused

The above error message is propagated up from the auto-attach supervisor service in the Supervisor cluster:

kubectl describe pkgi -n vmware-system-supervisor-services svc-auto-attach.vksm.broadcom.com

While connected to the Supervisor cluster context, the below configmap does not contain the full certificate chain intended for use with VMware Cloud Foundation Automation (VCFA):

kubectl get configmap -n vmware-system-mgmt-proxy image-registry -o jsonpath='{.data.trusted_certificates}'

In the Supervisor cluster, the kubectl-plugin-vsphere pod logs will show the following error:

[timestamp] [error] 7#0: *###### upstream SSL certificate verify error: (21:unable to verify the first certificate) while SSL handshaking to upstream, client: ##.##.##.##, server: mgmt-image-proxy.kube-system.svc.cluster.local, request: "GET /v2/ HTTP/1.1", upstream: "https://##.##.##.##:443/v2/", host: "mgmt-image-proxy.kube-system.svc.cluster.local"

NOTE: If a 502 Bad Gateway error occurs without an accompanying "unable to verify the first certificate" error, refer to the following documentation to ensure the proxy is correctly configured: Configuring the Supervisor Management Proxy

Environment

VMware Cloud Foundation Automation 9.0, 9.0.1, 9.0.2

vSphere Supervisor

Cause

VMware Cloud Foundation Automation (VCFA) did not properly propagate the full chain of its certificate to the Supervisor cluster.

A full chain could be comprised of the root, leaf and/or intermediate certificates.

This certificate mismatch prevents the auto-attach service from communicating with services on VCFA and pulling its required images.

Resolution

NOTE: The below workaround is only applicable for VCF 9.0, 9.0.1 and 9.0.2 environments.

If not available, collect and compose the full chain of the certificate in one pem file.
- Depending on the certificate used, this could be Leaf, Intermediate and Root certificates in one file.

Encode the full certificate chain:

cat <full certificate>.pem | base64 -w 0

SSH into the VCFA appliance VM as vmware-system-user

Establish root and Kubernetes access:

sudo su
export KUBECONFIG=/etc/kubernetes/admin.conf

Confirm that the vmsp-tls secret tls.crt is missing the full certificate chain:

kubectl get secrets vmsp-tls -n istio-ingress -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'

Take a backup of the vmsp-tls secret:

kubectl get secret vmsp-tls -n istio-ingress -o yaml > vmsp-tls-backup.yaml

Edit the vmsp-tls secret and change tls.crt to the full certificate chain that was encoded in Step 2:
```
kubectl edit secret vmsp-tls -n istio-ingress
```

Validate that the certificate was pushed to the VCFA database:

kubectl exec -n prelude vcfapostgres-0 -- \
  psql -U postgres -d tenantmanager -t -c \
  "SELECT pei.alias, cli.certificate
   FROM public_endpoint_info pei
   JOIN certificate_library_item cli ON pei.certificate_id = cli.id
   ORDER BY pei.alias;" 2>/dev/null

Trigger reconciliation and propagation to the Supervisor cluster:

for name in $(kubectl get supervisorregistries.vcd.addon.vcf.vmware.com -n prelude -o name); do
  kubectl annotate -n prelude "$name" \
    reconcile-trigger="$(date +%s)" --overwrite
done

If successful, it will return a message similar to the below:

supervisorregistry.vcd.addon.vcf.vmware.com/vcfa-service-manager-registry-<id>-<FQDN>-<supervisor name> annotated

To be thorough, re-install the VKS Cluster Management (VKSm) service from the VMware Cloud Foundation Automation | Provider Management web UI.
1. Navigate to Services - Overview and click on ACTIONS on the VKS Cluster Management module to Uninstall
2. Wait for the VKS Cluster Management supervisor service to uninstall from the Supervisor cluster
3. Click on ACTIONS to Install VKS Cluster Management again
4. When successful, the following Recent Tasks in VCFA will succeed and the VKSm service will return Healthy after a refresh:
```
Invoking notify Configuration(urn:vcloud:entity:broadcom:auto-attach-service:<id> for service-account-auto-attach-callback-service
Invoking notify Configuration(urn:vcloud:entity:broadcom:auto-attach-service:<id> for service-account-auto-attach-callback-service
Created Configuration Configuration(urn:vcloud:entity:broadcom:auto-attach-service:<id> for service-account-vcfa-service-manager
```