NSX Manager Inaccessible Due to Read-Only Disk Partition

Products

• VMware NSX

Issue/Introduction

Attempting to log into the NSX Manager appliances through SSH or console is blocked due to the presence of messages stating the Manager node is in read-only mode. 

• The NSX manager VMs were accessible via console from vCenter:
• The errors observed in the NSX GUI for these NSX Manager nodes are the same, or similar to: 
  

  Description
  
  The following disk partition on the Manager node {{entity_id}} is in read-only mode: {{disk_partition_name}}
  
  Recommended Action
  
  Examine the read-only partition to see if reboot resolves the issue or the disk needs to be replaced. Contact GSS for more information.
  
  Reported by Node
  
  <NSX manager node> 

• You may also see repeating messages similar to: 
  

  Starting default target
  
  You are in emergency mode
  
  sulogin: tcgetattr failed: Input/output error

Environment

NSX 3.x
NSX 4.x
NSX 9.x

Cause

File system corruption is most often caused when direct access to storage is lost.
After the storage issues are resolved, the manager appliance OS may choose to mount the file-system in read-only mode as a result of this corruption

Resolution

To resolve the issue perform the following actions: 

• Reboot the NSX manager and check if that resolves the issue and NSX manager becomes accessible and disk partition error on NSX manager gets resolved.
• If Reboot does not resolve the issue then perform below actions 

1. 1. Connect to the console of NSX manager appliance from vCenter 
   2. Reboot the appliance 
   3. When the GRUB menu appears, press the left SHIFT or ESC key quickly. Note: If you wait too long and the boot sequence does not pause, you must reboot the system again
   4. Keep the cursor on the Ubuntu selection
   5. Enter the user name root and the GRUB password for root (not the same as the appliance user root).
      Note: The default password is 'VMware1' before release 3.2, and 'NSX@VM!WaR10' for 3.2 and beyond.
   6. Search for the line starting with linux. At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 or Ctrl-X to boot the appliance. This should bring up the prompt to login with the appliance root user credentials.

Additional Information

For additional information refer to Manager node disk/partition is mounted as read-only alarm in NSX Manager - Disk Corruption correction using FSCK

Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).

• To contact Broadcom support for further assistance, visit the Contact Broadcom Support page.
• For creating and managing support cases, refer to Creating and Managing Broadcom Support Cases.