Unable to connect Cloud Gateway to Cloud vCenter Server

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
While in the Cloud Gateway HTML5 Client, attempts to Connect to Cloud vCenter Server fails with "Link failed with reason: Internal server error Contact support for further assistance"

Cloud Gateway - /var/log/vmware/hvc/hvc-svc.log

2019-10-21T17:44:34.792-04:00 [tomcat-exec-1  ERROR com.vmware.hvc.vapi.impl.LinksProviderImpl  opId=] Link failed with reason: Internal server error Contact support for further assistance
java.lang.Exception: Failed to create trust on the domain
        at com.vmware.hvc.setup.CertificateExchange.copyVcTrusts(CertificateExchange.java:472)
        at com.vmware.hvc.vapi.impl.LinksProviderImpl.createLinksV2(LinksProviderImpl.java:526)
.
.
.
Caused by: com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = vapi.bindings.method.impl.unexpected,
    defaultMessage = Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.Error,
    args = [com.vmware.vapi.std.errors.Error]
    [dynamic fields]: {
        localized = <unset>,
        params = <unset>
    }
}, LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.vcenter.trustmanagement.error,
    defaultMessage = Signing certificate does not allow digital signature use,
    args = [Signing certificate does not allow digital signature use]
    [dynamic fields]: {
        localized = <unset>,
        params = <unset>
 
    }
}],
    data = <null>
    [dynamic fields]: {
        error_type = INTERNAL_SERVER_ERROR
    }
}
        at com.vmware.vapi.std.errors.InternalServerError._newInstance(InternalServerError.java:152)

Environment

VMware vCenter Server 6.7.x
VMware vSphere ESXi 6.7

Cause

Pre-vSphere 5.5 certificates issued by RSA are carried over as Trusted Certificates, but are not used to sign tokens. When attempting to connect the Cloud Gateway to the Cloud vCenter Server, these certificates are unable to be pushed to form the trust denoted by "Failed to create trust on the domain."

Resolution

Before attempting, shut down all PSC/VC nodes (including the Cloud Gateway VM) and take powered-off snapshots. This is to ensure data integrity and prevent mid-flight replication amongst the PSCs.

1. Generate a New STS Signing Certificate on the Appliance - 
   https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.psc.doc/GUID-497233EA-AEF9-464B-A9C3-CCAEEA90C801.html
2. Refresh the Security Token Service Certificate - 
   https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.psc.doc/GUID-62981EA9-FEDD-4803-9CB6-29552FE703B1.html#GUID-62981EA9-FEDD-4803-9CB6-29552FE703B1
3. Once the new STS certificate has been generated/implemented, reboot the PSC.
4. Repeat the process (Steps 1-3) for any PSCs whose STS certificate is still issued by the old RSA
5. Old STS certificates issued by RSA will need to be removed via Jxplorer

Additional Information

Impact/Risks:
Unable to link Cloud Gateway to Cloud vCenter Server