One or more Manager nodes showing as degraded in the UI with multiple services down as seen after clicking the Details

Products

• VMware NSX

Issue/Introduction

 

• Connecting to the console of the degraded Manager(s) reveals error messages reporting failure to start one or more services

• The /var/log/vmkernel log file within the ESXi host where the Manager VM resided contains storage-related warning messages similar to the below:                                                                                 vmkwarning: CPU1:#### WARNING: StorageDeviceIO: ### : Device t10.NVMe_#######_###### performance has deteriorated. I/O latency increased from average value of ## seconds to 

Environment

VMware NSX 

Cause

The ESXi host where the Manager appliance resided had experienced a storage issue

 

Resolution

The following steps should help to restore connectivity to the NSX Manager node:

1. Migrate NSX manager node from the affected host 
2. Reboot the NSX manager node after the migration  

Additional Information

In case the the above steps do not resolve the issue, open support ticket through contacting Broadcom support 

To help expedite the process of resolving your issue, please gather logs from the affected manager appliances and upload them to us through the Broadcom support portal.  If you need assistance with the process of uploading logs to cases, please see the below linked KB article.
 https://knowledge.broadcom.com/external/article/140731/uploading-files-to-cases-on-the-broadcom.html