“Unable to connect to vCenter...” error when reconfiguring existing replications in Cloud Director Availability 4.x

Products

• VMware Cloud Director
• VMware vCenter Server

Issue/Introduction

• Reconfiguring existing replications fail in Cloud Director Availability, and you see the error:

Unable to connect to vCenter '########-####-####-####-############'.

• In the /opt/vmware/h4/cloud/log/cloud.log file on the destination Cloud Replication Management Appliance, you see entries similar to:

2021-05-19 13:08:53.781 ERROR - [UI-660650c3-54f0-41c6-bf98-3711187257aa-WY] [task-poller-3] com.vmware.h4.jobengine.JobExecution     : Task 67464ed3-6d87-4ac5-8d9d-f928fd0baf64 (WorkflowInfo{type='reconfigure', resourceType='vmReplication', resourceId='C4-########-####-####-####-############', isPrivate=false, resourceName='null’ }) has failed

com.vmware.h4.replicator.api.exceptions.FailedToAcquireVcConnection: Unable to connect to vCenter ‘########-####-####-####-############’.
        at jdk.internal.reflect.GeneratedConstructorAccessor2353.newInstance(Unknown Source)
        ...

• Similar entries to those present in the /opt/vmware/h4/cloud/log/cloud.log file can also be seen in the /opt/vmware/h4/manager/log/manager.log file on the destination Cloud Replication Management Appliance.
• In the /opt/vmware/h4/replicator/log/replicator.log file on the destination Cloud Replicator Appliance, you see entries similar to:

2021-05-19 11:08:53.731  INFO - [UI-660650c3-54f0-41c6-bf98-3711187257aa-WY-h6-VA] [job-46] c.v.identity.token.impl.SamlTokenImpl    : SAML token for SubjectNameId [[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from Element
2021-05-19 11:08:53.732  INFO - [UI-660650c3-54f0-41c6-bf98-3711187257aa-WY-h6-VA] [job-46] c.v.i.t.impl.X509TrustChainKeySelector   : Failed to find trusted path to signing certificate <OU=VMware,O=VMware,L=VMware,ST=VMware,C=DS,CN=STS>

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
...
2021-05-19 11:08:53.732 ERROR - [UI-660650c3-54f0-41c6-bf98-3711187257aa-WY-h6-VA] [job-46] c.v.identity.token.impl.SamlTokenImpl    : Signature validation failed

javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
...
2021-05-19 11:08:53.732 DEBUG - [UI-660650c3-54f0-41c6-bf98-3711187257aa-WY-h6-VA] [job-46] com.vmware.h4.replicator.vc.VcConnector  : Unable to connect to vCenter ‘########-####-####-####-############’.

com.vmware.vlsi.client.sso.SsoException: com.vmware.vim.sso.client.exception.MalformedTokenException: Signature validation failed
...

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue can occur when there are multiple service registrations in the Lookup service for the STS service.

Resolution

To verify you are experiencing this issue, query the Lookup service for the STS service registration.

1. Log in to the vCenter Server/PSC Appliance through SSH or console as root.
2. Change to Bash shell by typing shell and press Enter.
3. Run the following command to get the current sslTrust anchor stored for the vCenter Server:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
 

Example output:
Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: 810298bb-196f-4988-b9c3-2dbf796282bb
Site ID: default-first-site
Owner ID: [email protected]
Version: 2.0
Endpoints:
       Type: com.vmware.cis.cs.identity.sso
       Protocol: wsTrust
       URL: https://vc01.vsphere.local/sts/STSService/vsphere.local
-------------------------------------------------------
Service Product: com.vmware.cis
Service Type: cs.identity
Service ID: ec605cd2-52de-412f-b471-24bb3a47f2f6
Site ID: default-first-site
Owner ID: vc01.vsphere.local @vsphere.local
Version: 2.0
Endpoints:
       Type: com.vmware.cis.cs.identity.sso
       Protocol: wsTrust
       URL: https:// vc01.vsphere.local/sts/STSService/vsphere.local

Note: The expected result is one STS service entry per site.

If the service registrations have different Service IDs, contact VMware vCenter Server Support and note this Article ID (315112) in the problem description. For more information, see Creating and managing Broadcom support cases.