In an NSX-T Federation environment you are unable to view DFW statistics
search cancel

In an NSX-T Federation environment you are unable to view DFW statistics

book

Article ID: 322476

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

Symptoms:
  • You are using Federation and recently changed certificates on the NSX-T Managers.
  • You are unable to access the Local manager from the global manager UI.
  • Login works correctly on the local manager directly.
  • When you try to review the DFW (Distributed Firewall) hit count, you see the following error:
On local manager: "Error: No server's available: Error code 98"
On Global manager: "Error: No server's available: Error code 100"
  • On the Local Manager (LM) logs, we see the following error:
2022-11-10T12:46:28.251Z ERROR http-nio-127.0.0.1-7440-exec-17 NsxTRestClient 10989 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500018" level="ERROR" reqId="471aecb8-0518-441d-a05b-bce02a92c2e8" subcomp="manager" username="[email protected]"] Error in running REST api GET https://nsxt-mgr-1.corp.local:443/api/v1/firewall/sections/4ce169cb-a30b-4301-ac76-290f1ec54d85/rules/1000045/stats:The credentials were incorrect or the account specified has been locked.
2022-11-10T12:46:28.264Z ERROR http-nio-127.0.0.1-7440-exec-14 NsxTRestClient 10989 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM500018" level="ERROR" reqId="479101f2-b99e-489e-a077-3ba9e363db01" subcomp="manager" username="[email protected]"] Error in running REST api GET https://nsxt-mgr-1.corp.local:443/api/v1/firewall/sections/4ce169cb-a30b-4301-ac76-290f1ec54d85/rules/1000045/stats:The credentials were incorrect or the account specified has been locked.
2022-11-10T12:46:40.444Z  WARN http-nio-127.0.0.1-7440-exec-4 NsxTRestClient 10989 POLICY [nsx@6876 comp="nsx-manager" level="WARNING" reqId="5253074d-b5bb-40d7-893b-41e0a2f1a2d0" subcomp="manager" username="[email protected]"] Authentication failure with NSX manager, will retry
org.springframework.web.client.HttpClientErrorException$Forbidden: 403 : "{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}"
...
  • On the Global Manager (GM) logs we see the following errors:
2022-11-10T13:24:58.767Z  INFO http-nio-127.0.0.1-64440-exec-52321 UI_LOG 14654 - [nsx@6876 comp="global-manager" level="INFO" reqId="f74a34cb-76fa-4e58-b600-6229ac833110" subcomp="global-manager" username="[email protected]"] {"user":"","message":"Api Errors->","messageData":{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":400,"statusText":"OK","url":"https://nsx-gm-1.corp.local/global-manager/api/v1/global-infra/domains/default/security-policies/Temporary/rules/77ec11e0-5bc8-11ed-9f7a-c185615bbb94/statistics?enforcement_point_path=/global-infra/sites/HQ/enforcement-points/default","ok":false,"name":"HttpErrorResponse","message":"Http failure response for https://nsx-gm-1.corp.local/global-manager/api/v1/global-infra/domains/default/security-policies/Temporary/rules/77ec11e0-5bc8-11ed-9f7a-c185615bbb94/statistics?enforcement_point_path=/global-infra/sites/HQ/enforcement-points/default: 400 OK","error":{"details":"java.lang.NullPointerException","httpStatus":"BAD_REQUEST","error_code":100,"module_name":"common-services","error_message":"Error: General error has occurred. (Error code: 100)","error_data":{"status":400}}},"level":"Error","browser":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36 Edg/106.0.1370.42","time":"Thu Nov 10 2022 16:23:00 GMT+0300 (GMT+03:00)","location":"/app/security/dfw/module/categories/application"}


Environment

VMware NSX-T Data Center

Cause

This issue occurs due to how the federation Global manager imports new certificates for the local manager and leaves the old thumbprint in place on the Global manager, which causes communications issues between the Global managers and the Local managers.

Resolution

This issue is resolved in VMware NSX 3.2.3, available at VMware by Broadcom downloads.

Workaround:

  1. Browser to the LM site, find the SHA-256 Thumbprint.
    • In Chrome, other browsers may be different, this can be down by clicking the lock symbol in the address bar.
    • Selecting Connection is Secure and then clicking Certificate is Valid, you will be present with the certificate Thumbprints of the LM.
  2. It should look like:
    • 56 32 91 96 66 68 B9 C7 9F 05 40 A0 DA 0C E4 B3 AB 6C 23 8D 3E 1F A3 78 08 4A B7 CD 4D C7 B4 64 
  3. Replace all spaces with nothing, so it becomes:
    • 563291966668B9C79F0540A0DA0CE4B3AB6C238D3E1FA378084AB7CD4DC7B464
  4. Login to the GM and go to 'System->Location Manager'.
  5. Select 'Actions->Edit Settings' for relevant LM site, where we just got the Thumbprint for.
  6. Add the Thumbprint, without spaces, to SHA-256 Thumbprint field and the required password.
  7. Then click 'Check Version Compatibility' button in same box.
  8. Then click the 'Save' button.



▶
Powered by Wolken Software