After vCenter Certificates are replaced, Compute Manager in NSX-T is Unable to get Registered with the Option "Enable Trust" as Enabled
search cancel

After vCenter Certificates are replaced, Compute Manager in NSX-T is Unable to get Registered with the Option "Enable Trust" as Enabled

book

Article ID: 322036

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • Before replacing the vCenter certificate, the vCenter was registered as a Compute Manager in NSX Manager with the option "Enable Trust" enabled, and the registration was "UP".
  • After replacing the vCenter certificate, the Compute Manager shows "DOWN"
  • The procedure described in the following article to update vCenter's new certificate's thumbprint inside the compute manager in NSX-T has been followed:
          After vCenter Server certificate is replaced, compute manager connection is "Down" on NSX UI
Note: Before replacing the old certificate thumbprint with the new certificate thumbprint of vCenter in NSX Manager, please take a note of the old certificate's thumbprint of vCenter. Get the details on NSX Manager at: System > Fabric > Compute Managers > Edit the Compute Manager

           
  • With the option "Enable Trust" toggled on, the following error message comes up when attempting to save: 
            
    *Text like: Compute manager failed to get enabled as auth server due to error Thumbprint mismatch for" ... "Check that the thumbprint is correct.. Please check the hostname in the url is reachable from nsx. Error code: 90011)"
  • With the option "Enable Trust" toggled off, the compute manager connectivity comes back UP when saving. The "Enable Trust" option creates a 2 way trust between NSX-T Manager and vCenter. This feature is useful for services running in vCenter Server like Tanzu or vSphere Lifecycle Manager.
  • In NSX Manager logs at /var/log/proton/nsxapi.log there are error messages similar to:
2023-01-05T05:04:24.530Z ERROR http-nio-127.0.0.1-7440-exec-29 CmInventoryService 6190 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP90002" level="ERROR" reqId="3d285021-####-####-####-47bbcf789eda" subcomp="manager" username="admin"] Compute manager x.x.x.x failed to get enabled as auth server
  • Please take note of the new certificate's thumbprint of vCenter certificate.
    Note: Obtain the certificate using the following command:
    echo | openssl s_client -connect <vCenter IP or FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256
     
  • In NSX Manager logs at /var/log/proton/, search with the following pattern from the root CLI:
    grep "auth server details returned" nsxapi.log

    The expected results would be like the following:
2023-01-05T05:04:24.481Z  INFO http-nio-127.0.0.1-7440-exec-29 CmInventoryService 6190 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" reqId="3d285021-####-####-####-47bbcf789eda" subcomp="manager" username="admin"] compute manager <<Compute Manager's UUID>> auth server details returned CmAuthServerDetail{oidcUrl='https://<x.x.x.x>/openidconnect/vsphere.local/.well-known/openid-configuration', thumbprint='8d576a79##########################################', cmVersion='7.0.3'}
  •  Note the "thumbprint" value from the above output. If the obtained "thumbprint" value is matching the old value (noted earlier) from the vCenter certificate,  that means all the certificate components on vCenter were not properly replaced, and vCenter is still providing the old certificate's thumbprint for its 'openid-configuration' to NSX-T Manager.


If you are contacting Broadcom support about this issue, please provide the following:

  • NSX Manager log bundles.
  • vCenter log bundles related to the Compute Manager vCenter Server.
  • Text of any error messages seen in NSX GUI or command lines pertinent to the investigation.

Handling Log Bundles for offline review with Broadcom support:





Environment

VMware NSX-T Data Center

Cause

If vCenter's component certificates are not properly replaced it may still provide the old certificate's thumbprint for 'openid-configuration' which is required when "Enable Trust" option is toggled ON, while providing the new certificate's thumbprint when querying for it's HTTPS certificate, i.e.
echo | openssl s_client -connect <vCenter IP or FQDN>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256

Resolution

To fix the vCenter certificate issues please open up a support request with VMware vCenter support team.

Workaround:
While waiting for applying the permanent fix, to keep the compute manager connectivity as UP from NSX-T side, toggle OFF the "Enable Trust" option and save. Which will make the compute manager connectivity as UP.

You can use lsdoctor tool for fixing SSL trust mismatch in the vCenter server.  Using the 'lsdoctor' Tool
 
1. Upload the lsdoctor tool file to vCenter server.
 
2. Connect via ssh to vCenter server and extract the archived file.
 
3. Check the current status using the -l (lscheck) option.
>   python lsdoctor.py -l
 
4. If you can find any trust mismatch from the result of 3 then you can fix using -t (trustfix) option.
>   python lsdoctor.py -t
 
5. If running the 'trustfix' option, all services on vCenter need to be restarted. Use this command:
# service-control --stop --all && service-control --start --all
 
6. You can retry saving after all services are running properly.
 
Notes:
  • The 'lsdoctor -l' (lscheck) function can be run without any disruption of vCenter service.
  • If executing the trustfix (-t option), take an offline snapshot of the vCenter VM prior to executing it.
  • If is configured with vCenter High Availability (VCHA) enabled, it is recommended to remove VCHA and reconfigure it before taking the above steps.