Local Manager Information is not available in System overview UI due to API failure when CA-signed certificates are used

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Location Manager information cannot be viewed in the Global Manager UI
• RTEP calls to Local Manager fails
• When creating a segment and choosing a location the field may be uneditable.

Environment

VMware NSX-T Data Center

Cause

• Multiple certificates were installed in the global-manager or Certs transitioned from Self-signed to CA-Signed.
• Fetch the backup config information from GM to respective LM following class is being
  API call to the Global Manager that triggers the backup call 
  https://<GLOBAL_MANAGER_IP>/global-manager/api/v1/cluster/backups/overview?site_id=<SITE_ID>&show_backups_list=false&frame_type=LOCAL_MANAGER
• API call to get RTEP status fails
  https://<GLOBAL_MANAGER_IP>/global-manager/api/v1/ui-controller/overall-edge-clusters-rtep-status?site_id=<Site_ID>

Note: The Side_ID can be listed using the following API call:

GET <GLOBAL_Manager_IP>/global-manager/api/v1/global-infra/sites

 

Example:

https://<Global_manager_FQDN>/api/v1/ui-controller/overall-edge-clusters-rtep-status?site_id=494d6a89-####-####-####-8ffbb680050f

{
    "httpStatus": "BAD_REQUEST",
    "error_code": 513031,
    "module_name": "Policy",
    "error_message": "Error retrieving edge clusters RTEP status. Please contact the administrator."

From the Global Manager var/log/Syslog similar error messages are observed:

2021-05-14T17:22:33.696Z  INFO http-nio-127.0.0.1-64440-exec-2 NsxTrustManager 31858 SYSTEM [nsx@6876 comp="global-manager" level="INFO" reqId="427e914f-####-####-####-e6839b5a9120" subcomp="global-manager" username="<Username>"] Trust certificate for CN=<CN>,OU=<OU>,O=<O>,L=<L>,ST=<ST>,C=<C>
2021-05-14T17:22:33.733Z  INFO http-nio-127.0.0.1-64440-exec-2 RemoteEdgeClusterServiceImpl 31858 - [nsx@6876 comp="global-manager" level="INFO" reqId="427e914f-####-####-####-e6839b5a9120" subcomp="global-manager" username="<Username>"] Was not able to get data from remote site 494d6a89-####-####-####-8ffbb680050f. Error org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : [{"module_name":"common-services","error_message":"Internal server error has occurred.","details":"Client certificate not found in trust store","error_code":99}].
2021-05-14T17:22:33.733Z ERROR http-nio-127.0.0.1-64440-exec-2 RemoteEdgeClusterServiceImpl 31858 - [nsx@6876 comp="global-manager" errorCode="MP513031" level="ERROR" reqId="427e914f-63f1-4d4a-98f4-e6839b5a9120" subcomp="global-manager" username="<username>"] error while retrieving Edge clusters RTEP Status From all LM nodes

Resolution

Code fix available in NSX-T 3.1.3.7 and above.

Workaround:

This issue does not exist with self-signed certificates.