After vCenter Certificates are replaced, Compute Manager in NSX-T is Unable to get Registered with the Option "Enable Trust" as Enabled

Products

VMware NSX

Issue/Introduction

Before replacing the vCenter certificate, the vCenter was registered as a Compute Manager in NSX Manager with the option "Enable Trust" enabled, and the registration was "UP".
After replacing the vCenter certificate, the Compute Manager shows "DOWN"
The procedure described in the following article to update vCenter's new certificate's thumbprint inside the compute manager in NSX-T has been followed:
After vCenter Server certificate is replaced, compute manager connection is "Down" on NSX UI
Before replacing the old certificate thumbprint with the new certificate thumbprint of vCenter in NSX Manager, please take a note of the old certificate's thumbprint of vCenter. Get the details on NSX Manager at: System > Fabric > Compute Managers > Edit the Compute Manager
With the option "Enable Trust" toggled on, the following error message comes up when attempting to save:
NOTE: an error message similar to:
Compute manager failed to get enabled as auth server due to error Thumbprint mismatch for" ... "Check that the thumbprint is correct.. Please check the hostname in the url is reachable from nsx. Error code: 90011)"

With the option "Enable Trust" toggled off, the compute manager connectivity comes back UP when saving. The "Enable Trust" option creates a 2 way trust between NSX-T Manager and vCenter. This feature is useful for services running in vCenter Server like Tanzu or vSphere Lifecycle Manager.

In NSX Manager logs at /var/log/proton/nsxapi.log there are error messages similar to:

2023-01-05T05:04:24.530Z ERROR http-nio-127.0.0.1-7440-exec-29 CmInventoryService 6190 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP90002" level="ERROR" reqId="3d285021-####-####-####-47bbcf789eda" subcomp="manager" username="admin"] Compute manager x.x.x.x failed to get enabled as auth server

Please take note of the new certificate's thumbprint of vCenter certificate.
Note: Obtain the certificate using the following command:
```
echo | openssl s_client -connect <vCenter IP or FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256
```
Or removing it from the thumbprint window and save the settings. The resulting error will provide the newly seen thumbprint which you can past into the thumbprint section.

In NSX Manager logs at /var/log/proton/, search with the following pattern from the root CLI:
grep "auth server details returned" nsxapi.log

The expected results would be like the following:

2023-01-05T05:04:24.481Z  INFO http-nio-127.0.0.1-7440-exec-29 CmInventoryService 6190 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" reqId="3d285021-####-####-####-47bbcf789eda" subcomp="manager" username="admin"] compute manager <<Compute Manager's UUID>> auth server details returned CmAuthServerDetail{oidcUrl='https://<x.x.x.x>/openidconnect/vsphere.local/.well-known/openid-configuration', thumbprint='8d576a79##########################################', cmVersion='7.0.3'}

Note: the "thumbprint" value from the above output. If the obtained "thumbprint" value is matching the old value (noted earlier) from the vCenter certificate, that means all the certificate components on vCenter were not properly replaced, and vCenter is still providing the old certificate's thumbprint for its 'openid-configuration' to NSX-T Manager.

Environment

VMware NSX-T Data Center
VMware NSX

Cause

If vCenter's component certificates are not properly replaced it may still provide the old certificate's thumbprint for 'openid-configuration' which is required when "Enable Trust" option is toggled ON, while providing the new certificate's thumbprint when querying for it's HTTPS certificate, i.e.
```
echo | openssl s_client -connect <vCenter IP or FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256
```

Resolution

To fix the vCenter certificate issues please open up a support request with VMware vCenter support team. Please note this issue is specific to when enable trust is enabled, if the workaround of disabling trust mentioned in additional information does not restore connectivity, the compute manager disconnection is likely a different issue, see similar Issues in additional information.

Upload the lsdoctor tool file to vCenter server.
Connect via ssh to vCenter server and extract the archived file.
Check the current status using the -l (lscheck) option. (will not disrupt VC)
```
>   python lsdoctor.py -l
```
If you can find any trust mismatch from the result of 3 then you can fix using -t (trustfix) option.> python lsdoctor.py -t
If running the 'trustfix' option, all services on vCenter need to be restarted. Use this command: (Offline Snap shot of vCenter is recommended)
```
# service-control --stop --all && service-control --start --all
```
You can retry saving after all services are running properly.

Note:

If is configured with vCenter High Availability (VCHA) enabled, it is recommended to remove VCHA and reconfigure it before taking the above steps.
You can use lsdoctor tool for fixing SSL trust mismatch in the vCenter server. Using the 'lsdoctor' Tool

Additional Information

Workaround:

While waiting for applying the permanent fix, to keep the compute manager connectivity as UP from NSX-T side, toggle OFF the "Enable Trust" option and save. Which will make the compute manager connectivity as UP.

If you are contacting Broadcom support about this issue, please provide the following:

NSX Manager log bundles.
vCenter log bundles related to the Compute Manager vCenter Server.
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation.

Handling Log Bundles for offline review with Broadcom support:

Similiar issues:

KB - 376477 Error Reconnecting Compute Manager to NSX on SDDC-Managed Environment after Certificate Replacement on vCenter