Unable to login to vCenter when linked vCenters are down or unreachable

Products

• VMware vCenter Server

Issue/Introduction

• When linked vCenters are down or not reachable from the impacted vCenter, its services will not start.

• When logging into the impacted vCenter UI, the URL in the browser is switching to a linked vCenter for the login and comes back to vCenter URL.

• In vsphere_client_virgo.log, similar messages may be noted:

<timestamp> [INFO ] http-nio-5090-exec-6 70001628 100338 ###### com.vmware.identity.websso.client.Message Incoming or outgoing SAML message.
 Message Type:AUTHN_REQUEST
 ID:_<ID>
 SessionIndex:null
 Message source:https://<impacted_vcenter_fqdn>/ui/saml/websso/metadata
 Message destination:https://<linked_vcenter_fqdn1>/websso/SAML2/SSO/vsphere.local

• The following commands executed on the impacted vCenter does not list itself

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --ep-type com.vmware.cis.cs.identity.sso --no-check-cert | grep -A2 com.vmware.cis.cs.identity.sso

 Type: com.vmware.cis.cs.identity.sso
                Protocol: wsTrust
                URL: https://<linked_vcenter_fqdn1>/sts/STSService/vsphere.local
--
                Type: com.vmware.cis.cs.identity.sso
                Protocol: wsTrust
                URL: https://<linked_vcenter_fqdn2>/sts/STSService/vsphere.local
--

Environment

• VMware vCenter Server 7.x

Resolution

• Take an offline snapshot of all vCenter in ELM
  https://knowledge.broadcom.com/external/article/313886/vmware-vcenter-in-enhanced-linked-mode-p.html

When using multiple vCenter Server Appliances (VCSA) in the same Single Sign-on Domain, replicating in Enhanced Linked Mode (ELM), there is high potential of corruption of the domain if snapshots of the appliances are taken while they are in running state. Use of offline snapshots in ELM deployments is very strongly recommended for a safe rollback point. This means all appliances should be gracefully shut down, and snapshots need to be taken while the VCSAs are in powered off state (at the same time).

• Recreate entries on impacted vCenter with the following command

  • /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /root/machine.crt && /usr/java/jre-vmware/bin/java -cp /usr/lib/vmware-lookupsvc/lib/*:/opt/vmware/lib64/*:/usr/lib/vmware-sso/commonlib/*:/usr/lib/vmware/common-jars/*:.:* -Dlog4j.configurationFile=/usr/lib/vmware-lookupsvc/conf/initls-log4j2.xml -Dvmware.log.dir=/var/log/vmware/sso/ -XX:ErrorFile=/var/log/vmware/sso/hs_err_stsinstaller_pid%p.log -XX:HeapDumpPath=/var/log/vmware/sso/ com.vmware.vim.lookup.tools.InitializeLookupService --cert-path /root/machine.crt --host-name $HOSTNAME --http-port 443

• Restart vCenter services

  • service-control --stop --all && service-control --start --all