NSX Manager 9.1 upgrade shows complete from VCF Operations but a single Manager shows error "Failed to get <component> report - An unknown error has occurred"

Products

• VMware NSX

Issue/Introduction

• Upgrade of NSX Manager shows complete within VCF Operations and on the NSX Manager.
• No errors are reported within VCF Operations.
• A single NSX Manager does not load the UI correctly with the following errors displayed:
  
  An unknown error has occurred while retrieving the report metadata

Failed to get report - An unknown error has occurred

Unable to fetch details
  
  
• A notification in NSX stating "Failed to fetch system details...Error code: 513002" is present.
• Accessing the NSX UI via a different node shows none of these erorrs.
• The cluster status in the NSX UI on the System > Appliances page may show all nodes as healthy.
• NSX Manager syslog shows the following error:
  
  /var/log/syslog
[Timestamp] ERROR NSX 7756 [nsx@4413 comp="nsx-manager" errorCode="MP401" level="ERROR" logger="UserInfoUtil" msgID="SYSTEM" subcomp="manager" threadName="http-nio-127.0.0.1-7440-exec-7"] User ######## with groups [] and incoming roles null is not authorized to access API with rbac_feature search having required_permission read.
  
  
• NSX Manager policy-ui.log shows the following:
  
  /var/log/proton/policy-ui.log
[Timestamp] INFO NSX 8562 [nsx@4413 comp="nsx-manager" level="INFO" logger="UI_LOG" reqId="4f967b5d-####-####-####-7f5f10133b7b" subcomp="manager" threadName="http-nio-127.0.0.1-7440-exec-12" username="########"] {"user":"########","message":"Api Errors->","messageData":{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":400,"statusText":"OK","url":"https://<NSX Manager>/policy/api/v1/search?query=(<query>)&context=projects:/infra,/global-infra,/orgs/default/projects/default: 400 OK","error":{"details":"SEARCHING_EXCEPTION","httpStatus":"BAD_REQUEST","error_code":60506,"module_name":"nsx-search","error_message":"Error: Error occurred while fetching data. (Error code: 60506)","error_data":{"status":400}}},"level":"Error","location":"/app/system/home/overview/home/system/overview"}

Environment

VMware NSX

Cause

User/group information is not properly transmitted between manager nodes while the NSX 9.1 upgrade is still in progress.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

• This issue should not block or affect the upgrade.
• Continue with upgrade.
• When all NSX components are upgraded, the UI on the affected manager node will function as expected again.