NSX Manager is not working properly after experiencing storage issues affecting datastores related to the NSX appliance VMs

Products

VMware NSX

Issue/Introduction

After a storage outage including a VSAN failure or other issues with datastores backing NSX Manager VMs, any of the following may occur:

NSX Manager nodes may fail to boot up completely. When checking via console, there may be messages stating that partitions are in read-only (RO) status or services may fail to start with an error "Failed to start service X". Example screenshot below shows a failed service start in the console.
Access is unavailable to the UI for the specific impacted node. If multiple nodes were impacted then the GUI may be unreachable for all members and/or the VIP.
If UI access is available, the cluster may show that it is in a degraded status, and one or more of the Manager nodes may have alerts related to storage errors.
Home > Open Alarms may show Storage Error events with Critical Severity

Cause

An underlying storage layer outage or intermittent disk connectivity failure can result in filesystem corruption on the datastores backing the NSX appliance VMs. To protect data integrity, the Guest OS (Photon OS) remounts critical partitions in a read-only (RO) state. This prevents the NSX Manager and Edge nodes from writing to logs, updating the Corfu database, or maintaining the RabbitMQ message bus, ultimately leading to a loss of management plane services.

Resolution

In many cases, a reboot of an NSX Manager node can restore operability if the underlying storage issue has been resolved.
- If multiple nodes are impacted, reboot them one at a time and monitor the console as they come up.
- NSX will attempt to resolve filesystem issues. If it can, then check the status of services on the Manager node either under Fabric > Appliances > View Details (per node) or via admin CLI with command: get services
If rebooting does not correct the issue, refer to Manager node disk/partition is mounted as read-only alarm in NSX Manager Disk Corruption correction using FSCK for additional steps to attempt to recover filesystems mounted in read-only mode.
If all NSX Manager nodes are impacted, it is likely that a restoration from backup will be required.
- If only one node could not be recovered with above steps, an additional Manager appliance node can be added before or after deleting the faulted one.
  - Refer to Deploy NSX Manager Nodes to Form a Cluster from the UI for steps
  - To keep the original hostname and IP, delete the faulted one before adding the new one:
    1. Remove the faulted appliance from NSX
    2. Power off the VM in vSphere and delete it.
    3. Assign the original FQDN and IP when adding the new manager node.
  - Replace an NSX node in a VCF/SDDC Manager Environment has steps that can be referenced as well, whether the NSX is part of an SDDC Manager deployment or not.
If 2 out of 3 NSX Manager nodes are unhealthy, but one node can be confirmed to still be operational, the cluster can be deactivated on the healthy manager so it will remain alone as a single-node NSX Manager.
- Refer to NSX Manager cluster is DOWN or UNAVAILABLE if all nodes part of the the NSX Manager cluster is down or majority nodes are down
- After a "deactivate cluster" operation, Edge nodes may still show MPA Connectivity Down and be unaware of the new NSX Manager appliance configuration.
  - - A reboot of the Edge nodes will typically resolve this issue. It is recommended to first reboot the standby node(s) in Edge clusters and ensure they come up healthy before rebooting the active node. If this does not resolve the issue, open a case with Broadcom support.
- After ensuring the single NSX Manager node is running all services properly (at this point the UI for that node would be available), add two nodes ("appliances") to form a 3-node cluster again.
  - Deploy NSX Manager Nodes to Form a Cluster from the UI to add additional nodes that replace the prior cluster members.
  - Refer to (KB 317159) When reconfiguring NSX Manager node in NSX-T Data Center, Deployment fails with the "Error code: 21016" if an error message indicates that a new node with the same name cannot be added even after the node has been deleted from disk in vCenter.
In case the above options do not work a backup restoration will be needed.
- Follow the process in the Broadcom Techdoc, Restore a Backup to deploy a single NSX Manager Appliance VM, then Deploy NSX Manager Nodes to Form a Cluster from the UI to add additional nodes that replace the prior cluster members.
- Note that the backup restoration is only done on a single node, and additional nodes sync data from the one that was restored when they are added to the cluster.

Additional Information

If you are contacting Broadcom support about this issue, please provide the following:

NSX Manager support bundles (if possible)
Text of any error messages seen in NSX GUI or command lines pertinent to the investigation.

Handling Log Bundles for offline review with Broadcom support