EAM service not authenticated to VC after VCSA upgrade to 7.0 U1 or later

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCLS VM(s) may not be deployed after vCenter Server Appliance Upgrade to 7.0 U1
• If you are using NSX, you may have issues with NSX authenticating to EAM, impacting host preparation and service insertion. 
• You may see the below in /var/log/vmware/eam/eam.log:

2020-11-12T12:55:13.265Z | INFO | sts-0 | Workflow.java | 121 | [CreateSAMLToken:18620e066128741f] FAILED
com.vmware.eam.sso.exception.TokenNotAcquired: Couldn't acquire token due to: The SSL certificate of STS service cannot be verified
 at com.vmware.eam.sso.impl.AcquireTokenAdapter.handleException(AcquireTokenAdapter.java:69) [eam-server.jar:?]
 at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$AsyncCommand.call(SecurityTokenServiceImpl.java:1168) [wstClient.jar:?]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: com.vmware.vim.sso.client.exception.CertificateValidationException: The SSL certificate of STS service cannot be verified
 at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:975) ~[wstClient.jar:?]

Environment

VMware vCenter Server 7.0.x

Cause

• From EAM logs it can be observed that STS endpoint used is https://<vcenter fqdn/ip>/sts/STSService/vsphere.local .

• EAM doesn't log STS Certificate at INFO logging level. For more information , see Increasing VMware vCenter Server and VMware ESX/ESXi logging levels (1004795)

• However, you can observe some lookup-service log which has the STS service Certificate, but  cannot be verified.

Resolution

There may be an issue with the STS Certificate in the service registration. Run the lsdoctor script to fix the issue
 

Installation

• To use lsdoctor, you must download the ZIP file attached to this article.  Then, use the file-moving utility of your choice (WinSCP for example) to copy the entire ZIP directory to the node on which you wish to run it.

NOTE:  If you have troubles connecting to a vCenter appliance using WinSCP, please see Error when uploading files to vCenter Server Appliance using WinSCP

• Once the tool is copied to the system, Change your directory to the location of the file, and run the following command:

    unzip lsdoctor.zip

 

NOTE:  When running the tool, be sure you are currently in the “lsdoctor-master” directory.
 

Launching the Tool

• First, ensure you are in the lsdoctor-master directory from a command line.

To run lsdoctor, use the following command:

#python lsdoctor.py --help

1.Run lsdoctor with the "-t, --trustfix" option to fix any trust issues.
       #python lsdoctor.py -t

2.Restart all vCenter services
      #service-control --stop --all
      #service-control --start --all

3.If still vCLS VMs still not getting deployed , run lsdoctor with the "-r, --rebuild" option to rebuild service registration.
     #python lsdoctor.py -r

Additional Information

For more information about lsdoctor, see  Using the 'lsdoctor' Tool (80469)

Impact/Risks:

WARNING

Before using lsdoctor to make any changes, ensure you have taken proper snapshots of your SSO domain. This means that you must shut down all VCs that are in the SSO domain at the same time, then snapshot them, and power them on again.  If you need to revert to one of these snapshots, shut all the nodes down, and revert all nodes to the snapshot. Failure to perform these steps will lead to replication problems across the VCs databases.

LIMITATIONS

Currently, lsdoctor supports vCenter 6.5 and above. When new builds of vCenter are released, lsdoctor must be updated asynchronously.  This means lsdoctor support for the latest version of vCenter may be updated sometime after a new build is released.