After onboarding new location to Global Manager, appliances are showing as not available

Products

• VMware NSX

Issue/Introduction

• After adding a new location to the global manager in a federated environment, the location is presented as Not Available in the UI:
  GM UI: System > Location Manager
  
  
  
  • Appliances: Not Available - Try again later
  • Latest Backup: Not Available - Try again later
  • Automatic Backups: Not available - Try again later
  • Remote Tunnel Endpoint Status: Not available - Try again later
• Sync Status may be in Success status.
• The location will show a red banner with the error:
  Default transport zone is not found of Local manager at site <LM-SITE name>. Please wait for default transport zone discovery or reload the enforcement point, and entry.
• Trying to remove the location is unsuccessful and will present a red banner with the error:
  Error: Site is not reachable because the Site or its services are down or GM certificate is unavailable on LM. (Error code: 500258)
• The Local Manager site certificates on the Global Manager and the Global Manager site certificates on the local managers are missing:
  GM UI: System > Settings > Certificates
  LM UI:  System > Settings > Certificates
  
  • These site certificates have the the category Remote Site
• You have verified that the thumbprint for the Location is not incorrect, by following KB: Unable to fetch TransportZoneListResultDto List from NSX, enforcement point default observed on Global Manager (Error code: 500139) 
• You have confirmed that the required TCP ports 1236 and 443 are open and communication is possible between GM to  all LM  appliances and LM to all GM appliances:

  Example using curl as root user:

  • GM to LM:
curl -v telnet://<LM appliance IP>:443
* Connected to <IP> port 443
curl -v telnet://<LM appliance IP>:1236
* Connected to <IP> port 1236

    LM to GM: 
    curl -v telnet://<GM appliance IP>:443
* Connected to <IP> port 443
curl -v telnet://<GM appliance IP>:1236
* Connected to <IP> port 1236
• Log lines similar to the below are encountered on the NSX Global Manager in /var/log/gmanager/gmanager.log 

  <TIMESTAMP> ERROR http-nio-127.0.0.1-64440-exec-29 NapiBackupGenerationServiceImpl 79929 SYSTEM [nsx@6876 comp="global-manager" errorCode="MP29256" level="ERROR" reqId="<UUID>" subcomp="global-manager" username="admin"]
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "{"module_name":"common-services","error_message":"Internal server error has occurred.","details":"Client certificate not found in trust store","error_code":99}"
        at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170) ~[spring-web-5.3.34.jar:5.3.34]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

The Global Manager and Local Manager site certificates were not exchanged after onboarding the Location, preventing further communication to establish the correct status.

Resolution

Run the CARR script on all Local Manager appliances and Global Manager appliances. Ensure that SSH is allowed between the GM and LM sites:
Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX

After this verify the site certificates have successfully been exchanged: 

• GM UI: System > Settings > Certificates
• LM UI:  System > Settings > Certificates

Note: These site certificates have the the category Remote Site

Additional Information

Unable to fetch TransportZoneListResultDto List from NSX, enforcement point default observed on Global Manager (Error code: 500139) 
NSX - VMware Ports and Protocols
Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX