NSX Manager vm console reports "Login incorrect" or "Failed to conect lvmetad" during power on/reboot

Products

• VMware NSX

Issue/Introduction

• 2 of the 3 manager nodes are stuck at boot with "Login incorrect" error and NSX UI isn't accessible via any of the manager IP's or the VIP too.
• NSX Manager console in vSphere reports "Login incorrect" during boot
• Below is the screenshot of the error
  
  
  
  
• When you run 'get cluster status' command from the working node, all the services of the all the 3 manager nodes are either unavailable or in unknown status
  
  Below is the screenshot of the command output
  
  
  
  
• When you run 'get cluster config' command on the working node, you see an error stating 'The get cluster config operation cannot be processed currently, please try again later'
  
  

Environment

VMware NSX

Cause

The issue occurs if there are filesystem corruption on NSX Manager nodes.
This can be due to an underlying storage issue. 

 

Resolution

• Investigate if there are any underlying storage issues in the environment and fix those issues
• Reboot the problem NSX manager nodes and monitor to see if reboot of the nodes, fixes the issue.
• If the issue persists even after reboot of the manager nodes and if there are no storage issues, please follow the below KB to fix the disk/partition is mounted as read-only issue.
  Manager node disk/partition is mounted as read-only