[VMC on AWS] Hybrid Link Mode is broken post upgrading SDDCs to version 1.20.x

Products

• VMware Cloud on AWS

Issue/Introduction

Provides steps for identification and remediation. 

Symptoms:

Symptom 1: 

• Post upgrading SDDCs to version 1.20, Cloud vCenter is not visible in the HLM UI to be managed.

• You may see error "One or more vCenter Server systems are incompatible with the vSphere Web client https:/vCenter.sddc-x-xxx-xxx-xxx.xxxxxxxx.com:443/sdk"

Symptom 2: 

• Initital vCGA setup fails ar Step 4 of selecting admin group with error "Error: Link failed with reason: Internal system error: com.vmware.vim.vmomi.client.exception.InvalidSslCertificateException: Invalid SSL certificate (HTTP 526 status code)"

Cause

• SDDC M20 bundle upgrades vCenter Server on the VMC to version 8.x. vCenter 8.x does not support client connections presenting SHA1 signature thumbprint.Any Cloud Gateway appliance using SHA1 certificates will not be able able connect to the VMC vCenter.

• For some gateways which do not have SHA1 certificates, might be out of sync with respect to intermediate/root certificate on VMC vCenter. The HTTP status code 526 occurs when a HTTP connection cannot be made to the origin server due to an invalid SSL certificate.The issue occurs if the gateway does not have the Trusted roots required to verify the server cert presented by the cloud vCenter. The cloud vCenter is issued by intermediate CA cert

 

Resolution

Validate if there are any expired or weak signature algorithm certificates in On-premise vCenter trusted root store and un-publish them. Please follow steps in the KB2146011  to identify and un-publish expired/invalid certificates and try re-linking the Cloud Gateway. 

If it still fails during the setup or you are still unable to see the VMC vCenter, Open a Support Request with VMware Global Support Services and include the below required information. 

1. Collect the On-premise vCenter logs and the Cloud gateway Logs. 
2. Run below command to dump VECS on Cloud Gateway and upload the dump to the Support Request

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i ; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text ; done;

Additional Information

Impact/Risks:
Single pane of glass experience to manage linked vCenters will be unavailable. Other functionalities are not affected. HLM will be broken.