Pre-upgrade check for clusters fail with Airgap Server not Healthy or Certificate expired

Products

• VMware Telco Cloud Automation

Issue/Introduction

You may encounter an issue where pre-upgrade checks for their clusters fail with an error message "Airgap Server not Healthy or Certificate expired."

Environment

3.x

Cause

This issue occurs when the customer initially use the Airgap server certificate as the input for TCA and subsequently modifies it to include the certificate chain resulting in pre-upgrade check failures.

Error: Airgap Server not Healthy or Certificate expired

"Message:
Airgap Server not Healthy or Certificate expired.
{"airgapHealth": "", "airgapCertificateExpiry": "2034-00-00 00:00:00",
"message": "error executing curl command to reach airgap server. Please check for cluster connectivity to airgap server", "Status": ""}.
Skipping further upgrade checks..."

Resolution

The tca-diagnosis-operator in TCA is responsible for diagnostic checks on TKGm clusters. The telco.vmware.com/airgap-ca-cert annotation on a TcaKubernetesCluster resource provides the necessary CA certificate for this secure
communication. The tca-diagnosis-operator reads this annotation to obtain the certificate. Validate if the value of the annotation 'telco.vmware.com/airgap-ca-cert' has the correct certificate.

1. Verify the certificate used by the management cluster 

kubectl get tkc <management_cluster> -n <management_cluster>  -o jsonpath='{.metadata.annotations.telco\.vmware\.com/airgap-ca-cert}' | base64 -d | openssl x509  -noout -dates

     2. Obtain the correct root certificate from the Airgap server (/etc/docker/certs.d/{$AIRGAP_FQDN}:8043/ca.crt)

     3. Update tkc cr with correct certificate 

kubectl patch tkc <management_cluster> -n <management_cluster> --type='merge' -p "{\"metadata\":{\"annotations\":{\"telco.vmware.com/airgap-ca-cert\":\"$(cat ca.crt | base64 -w0)\"}}}"