NSX cannot connect to SSO in an environment with multiple PSCs or STS certificates after installing or upgrading to NSX-V 6.4.2

Products

• VMware vCenter Server
• VMware NSX Data Center for vSphere

Issue/Introduction

• NSX is not able to connect to the vCenter SSO server after deployment of NSX 6.4.2 when:
• Configuring the SSO Lookup Service fails with the following error in vsm.log:

 

2018-08-28 17:28:53.178 CEST INFO TaskFrameworkExecutor-2 X509TrustChainKeySelector:190 - Failed to find trusted path to signing certificate <CN=[SSO hostname]>
java.security.cert.CertPathBuilderException: Unable to find certificate chain.
 

• At the NSX Manager Virtual Appliance Management Webpage, you will observe the following error under “Manage vCenter registration option:
   

NSX Management Service operation failed. ( Initialization of Admin Registration Service Provider failed. Root Cause: Signature validation failed )
Lookup Service https://####.example.com:443/lookupservice/sdk presented an SSL certificate with the following thumbprint:
##:##:##:##:##:##:##:##:DE:AD:BE:AF:54:56:5B:##:##:##:##:##
Proceed with this certificate?

Environment

VMware NSX Data Center for vSphere 6.4.x

Multiple PSCs are involved or an embedded PSC with multiple certificate chains are involved

Cause

This occurs when you have multiple trusted chain certificates, because the upgraded NSX client code uses only the first chain to configure the trust store.
 

Note -  that multiple chains can come in play when there are multiple PSC nodes with different Tenant Credentials. This can be hit when vCenter has been upgraded from a previous version where different PSCs had different signing credentials. To support that scenario, we need to include chains from both issuers, to allow the token to be validated against PSC nodes with different signing credentials.

Resolution

This issue is resolved in VMware NSX for vSphere 6.4.3.

Workaround:
To work around this issue if you are not able to upgrade: 

A script has been developed that replaces the JAR file in the NSX manager. The workaround requires a signed script to be executed using REST API call to NSX Manager.

1. Download the attached PscAndNetXFix.encoded file.
2. Run the following POST call on NSX Manager via one of the two options below.
   
   
   Option 1: Postman
   Method: POST
   URL: https://<nsxmgr_ip>/api/1.0/services/debug/script
   Authentication: Basic authentication (Username: admin)Expected Response: 200
   Headers: content-type - application/xml
   Body: copy contents of the attached file PscAndNetXFix.encoded
    
   Note: During copy/paste of the contents into the body, make sure no extra line/characters get added at the end in order to have the API run successfully. The content of the PscAndNetXFix is roughly 11 MB, opening the file, copying the content and running the API call all take time to process. Do not interrupt the API call. Proceed to Step #3 only if the response is 200. 
    
   Option 2: CURL (Run from your local machine or the node that contains the PscAndNetXFix.encoded file)  
   curl -k -X POST -H "Content-Type: application/xml" -d "@PscAndNetXFix.encoded" -u user:password https://<nsxmgr_ip>/api/1.0/services/debug/script
   
   
3. After running the API, restart the NSX management service in the NSX UI.
4. Once NSX management service has started, re-register the lookup service in NSX UI.

Note: If the above workaround fails, contact Broadcom support.