VCF 9.1.0 - Supervisor Services Error - "unexpected status code 502 Bad Gateway"

Products

VMware vSphere Kubernetes Service VCF Automation

Issue/Introduction

One or more supervisor services are stuck in Error Status and the associated Signature shows as Unavailable - Signature Certificate is not available.

When viewing details of the affected supervisor service from the vSphere web UI, the following error is shown:

Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: GET https://mgmt-image-proxy.kube-system.svc.cluster.local/v2/: unexpected status code 502 Bad Gateway: <html><head><title>502 Bad Gateway</title></head><body><center><h1>502 Bad Gateway</h1></center><hr><center>nginx/version</center></body></html>Get "http://mgmt-image-proxy.kube-system.svc.cluster.local/v2/": dial tcp <mgmt-image-proxy external IP>:80: connect: connection refused

The above error message is propagated up from the ReconcileFailed supervisor service in the Supervisor cluster:

kubectl describe pkgi -n vmware-system-supervisor-services <supervisor service pkgi name>

While connected to the Supervisor cluster context, the below configmap does not contain the full certificate chain intended for use with VMware Cloud Foundation Automation (VCFA):

kubectl get configmap -n vmware-system-mgmt-proxy image-registry -o jsonpath='{.data.trusted_certificates}'

Environment

VMware Cloud Foundation Automation 9.1.0

vSphere Supervisor

Cause

VMware Cloud Foundation Automation (VCFA) did not properly propagate the full chain of its certificate to the Supervisor cluster.

A full chain could be comprised of the root, leaf and/or intermediate certificates.

An internal process fetches the full certificate chain to build the payload for VCF Automation and propagate the certificate chain to the Supervisor cluster.

During this process, the payload incorrectly assigns only the leaf certificate to the "cert" component and only the root certificate to the "ca" component, effectively stripping away any intermediate certificates.

This incomplete certificate on both VCFA and the Supervisor cluster prevents the affected supervisor services from communicating with services on VCFA.

As a result, restarts and new supervisor service deployments dependent on integration with VCFA will fail.

Resolution

NOTE: The below workaround is only applicable for VCF 9.1.0 environments.

If not available, collect and compose the full chain of the certificate in one pem file.
- Depending on the certificate used, this could be Leaf, Intermediate and Root certificates in one file.

Encode the full certificate chain:

cat <full certificate>.pem | base64 -w 0

SSH into the VCFA appliance VM as vmware-system-user

Establish root and Kubernetes access:

sudo su
export KUBECONFIG=/etc/kubernetes/admin.conf

Confirm that the ingress-vcfa-tls-external secret's tls.crt is missing the full certificate chain:

kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d

kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text

Take a backup of the ingress-vcfa-tls-external secret:

kubectl get secret -n prelude ingress-vcfa-tls-external -o yaml > ingress-vcfa-tls-external-backup.yaml

Edit the ingress-vcfa-tls-external secret and change tls.crt to the full certificate chain that was encoded in Step 2:
```
kubectl edit secret -n prelude ingress-vcfa-tls-external
```

Confirm that the secret was updated with the full certificate chain:

kubectl get secret -n prelude ingress-vcfa-tls-external -o jsonpath='{.data.tls\.crt}' | base64 -d

Initiate a sync and immediate reconciliation, pushing the updated certificate chain to the Supervisor cluster:
```
for name in $(kubectl get supervisorregistries.vcd.addon.vcf.vmware.com -n prelude -o name); do
  kubectl annotate -n prelude "$name" \
    reconcile-trigger="$(date +%s)" --overwrite
done
```
This command updates the timestamp of this supervisorregistries custom resource to the current date.
It can take up to 10 minutes to propagate to the Supervisor cluster.
Confirm that the Supervisor services in the vSphere web UI now show Configured instead of Error.
If the error persists, check that the certificate chain was formatted correctly.

You can validate from the Supervisor cluster context that the certificates propagated successfully:

kubectl get configmap -n vmware-system-mgmt-proxy image-registry -o jsonpath='{.data.trusted_certificates}'

It is not necessary to manually edit this configmap object.

VCFA should be pushing the certificates based on the contents in the previously updated ingress-vcfa-tls-external secret object.

Additional Information

VCFA routinely syncs and propagates its certificate configuration from VMSP to the Supervisor cluster every 3 hours.