Error Code: 500016 when trying to view group members or checking the DFW rule status on the Global Manager (Error: I/O error)

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

• NSX Federated environment.
• Recently replaced certificates or expired certificates on Local Manager and/or Global Manager.
• You see an error stating General Error on one ore more Local Managers on the Location Manager page on the Global Manager.
• When trying to view members on the global manager you see similar errors in the NSX UI:

Error: I/O error on GET request for "https://<nsx-mgr-ip>/policy/api/v1/global-infra/domains/default/groups/ipset-########-####-####-####-########/members/ip-addresses": UUID; nested exception is javax.net.ssl.SSLHandshakeException: UUID (Error code: 500016)

Note: The error above is not exclusively seen for IP Addresses, the same error can be seen for Virtual Machines, NSX Segments, etc...

• Also, when trying to check the status for the DFW rules in the Global NSX UI, you may get this unknown status for the problematic LM:  

• Same, if you try to check the rule statistics:

Additional symptom:

• When selecting 'Generate BGP Summary,' we are receiving the following error:

  Error: I/O error on GET request for "https://####-########.#######.com/policy/api/v1/global-infra/tier-0s/NA-##-##-###-###-Exit/locale-services/NA-S2-####-########.#######.com/bgp/neighbors/status": #######################################################; nested exception is javax.net.ssl.SSLHandshakeException: ####################################################### (Error code: 500016)

Environment

VMware NSX-T Data Center

Cause

1. After replacing a certificate, the thumbprint for the site locations may not get automatically replaced.
2. Expired certificates are seen on the Local Managers

Resolution

• Verify the thumbprint is correct for both Local Manager sites.

  • This can be checked on the Global Manager, System Location Manager, under the site location Actions > Edit Settings and Check Version Compatibility.
    • If it fails the thumbprint may need to be updated for both locations.
      
      
  • The thumbprint can be collected by logging to the VIP assigned local manager on to each local manager site and running the command 'get certificate cluster thumbprint' in admin mode.
  • Copy the thumbprint and paste it in the the Thumbprint text box under the site location Actions > Edit Settings.
    
  • Click on "Check Version Compatibility" again and make sure it succeeds.

Additional Information

References:

• Add a Location
• Replace Certificates
  
  

Impact/Risks:

• Cannot view group members on the Global Manager.
• Local manager sites are not synced with the Global Manager.