Unable to access nsx manager due to disk-partition on the manager node is in read-only mode

Products

• VMware NSX

Issue/Introduction

After resolving an underlying storage issue on the setup, some of the NSX managers became inaccessible as the disk-partition on the manager node is in read-only mode. The NSX manager VMs were accessible via console on vcenter. 

The error/alarm observed on NSX GUI for these NSX Manager nodes were as below  

Description

The following disk partition on the Manager node {{entity_id}} is in read-only mode: {{disk_partition_name}}

Recommended Action

Examine the read-only partition to see if reboot resolves the issue or the disk needs to be replaced. Contact GSS for more information.

Reported by Node

<NSX manager node> 

Environment

NSX 3.x

NSX 4.x

Cause

The issue could have occurred due to file system corruption and that could have happened when storage issue was observed on the setup. 
After the storage issues were resolved, the manager appliance OS might choose to mount the file-system in read-only mode as the file system could have been corrupted.

Resolution

To resolve the issue perform below actions 

Reboot the NSX manager and check if that resolves the issue and NSX manager becomes accessible and disk partition error on NSX manager gets resolved. 

If Reboot does not resolve the issue then perform below actions 

• Connect to the console of NSX manager appliance from vcenter 
• Reboot the appliance 
• When the GRUB menu appears, press the left SHIFT or ESC key quickly. Note: If you wait too long and the boot sequence does not pause, you must reboot the system again
• Keep the cursor on the Ubuntu selection
• Enter the user name (root) and the GRUB password for root (not the same as the appliance user root). Note: The default password is 'VMware1' before release 3.2 and 'NSX@VM!WaR10' for 3.2 and beyond.
• Search for the line starting with linux. At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 or Ctrl-X to boot the appliance. This should bring up the prompt to login with the appliance root user credentials.

Additional Information

For additional information refer KB 330478

Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).