NSX-T Federation Issues: DFW Statistics Not Visible and Local Manager UI Inaccessible via Quick Change Site Manager
search cancel

NSX-T Federation Issues: DFW Statistics Not Visible and Local Manager UI Inaccessible via Quick Change Site Manager

book

Article ID: 322476

calendar_today

Updated On:

Products

VMware NSX VMware vDefend Firewall

Issue/Introduction

  • Recent upgrade of Global and Local Managers
  • You are using Federation and recently changed certificates on the NSX-T Managers.
  • You are unable to access the Local manager from the global manager UI.
  • Login works correctly on the local manager directly.
  • When checking thumbprint in the Global Manager Location Manager for each of the Local Manager sites receive: Error Thumbprint <Thumbprint ID> is not matching with that of IP/FQDN <FQDN/IP-of-site>. (Error code: 530031)
  • When you try to review the DFW (Distributed Firewall) hit count, you see the following error:
On local manager: "Error: No server's available: Error code 98"
On Global manager: "Error: No server's available: Error code 100"
  • On the Local Manager (LM) logs, we see the following error:
2022-11-10T12:46:28.251Z ERROR http-nio-127.0.0.1-7440-exec-17 NsxTRestClient 10989 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500018" level="ERROR" reqId="########-####-####-########c2e8" subcomp="manager" username="<nsx-user>"] Error in running REST api GET https://<nsx-manager-FQDN/IP>:443/api/v1/firewall/sections/########-####-####-############/rules/1000045/stats:The credentials were incorrect or the account specified has been locked.
2022-11-10T12:46:28.264Z ERROR http-nio-127.0.0.1-7440-exec-14 NsxTRestClient 10989 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500018" level="ERROR" reqId="########-####-####-########db01" subcomp="manager" username="<nsx-user>"] Error in running REST api GET https://<nsx-manager-FQDN/IP>:443/api/v1/firewall/sections/########-####-####-############/rules/1000045/stats:The credentials were incorrect or the account specified has been locked.
2022-11-10T12:46:40.444Z  WARN http-nio-127.0.0.1-7440-exec-4 NsxTRestClient 10989 POLICY [nsx@6876 comp="nsx-manager" level="WARNING" reqId="########-####-####-########a2d0" subcomp="manager" username="<nsx-user>"] Authentication failure with NSX manager, will retry
org.springframework.web.client.HttpClientErrorException$Forbidden: 403 : "{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}"
...
  • On the Global Manager (GM) logs we see the following errors:
2022-11-10T13:24:58.767Z  INFO http-nio-127.0.0.1-64440-exec-52321 UI_LOG 14654 - [nsx@6876 comp="global-manager" level="INFO" reqId="########-####-####-########3110" subcomp="global-manager" username="<nsx-user>"] {"user":"","message":"Api Errors->","messageData":{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":400,"statusText":"OK","url":"https://<nsx-manager-FQDN/IP>/global-manager/api/v1/global-infra/domains/default/security-policies/<security-policy-name>/rules/########-####-####-############/statistics?enforcement_point_path=/global-infra/sites/<site-name>/enforcement-points/default","ok":false,"name":"HttpErrorResponse","message":"Http failure response for https://<nsx-manager-FQDN/IP>/global-manager/api/v1/global-infra/domains/default/security-policies/<security-policy-name>/rules/########-####-####-############/statistics?enforcement_point_path=/global-infra/sites/####/enforcement-points/default: 400 OK","error":{"details":"java.lang.NullPointerException","httpStatus":"BAD_REQUEST","error_code":100,"module_name":"common-services","error_message":"Error: General error has occurred. (Error code: 100)","error_data":{"status":400}}},"level":"Error","browser":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.42","time":"Thu Nov 10 2022 16:23:00 GMT+0300 (GMT+03:00)","location":"/app/security/dfw/module/categories/application"}
  • When using the quick change Site Manager feature to change to a Local Manager instance, the Local Manager UI will load, however unable to navigate within the Local Manager UI window.
    When checking Browser Developer Tools Interface (F12)-->Network Tab, full-sync-states times out and fails
    Local Manager UI when Navigating from Global Manager Quick Change:

Environment

VMware NSX

 

Cause

This issue occurs due to how the federation Global manager imports new certificates for the local manager and does not update the old thumbprint in place on the Global manager, which causes communications issues between the Global managers and the Local managers.

Resolution

Workaround:

Update the Thumbprint on the Global Manager:

  1. Browser to the LM site, find the SHA-256 Thumbprint.
    • In Chrome, other browsers may be different, this can be down by clicking the lock symbol in the address bar.
    • Selecting Connection is Secure and then clicking Certificate is Valid, you will be present with the certificate Thumbprints of the LM.
    • It should look like: 56 32 91 96 66 68 B9 C7 9F 05 40 A0 DA 0C E4 ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## ## 
  2. Replace all spaces with nothing, so it becomes:

        563291966668B9C79F0540A0DA0CE4##################################

  3. Login to the GM and go to 'System->Location Manager'.
  4. Select 'Actions->Edit Settings' for relevant LM site, where we just got the Thumbprint for.
  5. Add the Thumbprint, without spaces, to SHA-256 Thumbprint field and the required password.
  6. Then click 'Check Version Compatibility' button in same box.
  7. Then click the 'Save' button.
▶
Powered by Wolken Software