Manager node disk/partition is mounted as read-only alarm in NSX Manager

Products

• VMware NSX

Issue/Introduction

Title: Alarm for Manager disk/partition is mounted as read-only
Event ID: manager_health.storage_error

Alarm Description

• Purpose: Indicates whether a Manager disk/partition is mounted as read-only
• Impact: Some functionality will not work if any of the manager partition gets mounted as read-only (like - login error, disk write error)

Environment

VMware NSX-T Data Center

Resolution

Steps to Resolve
For 3.0.2 and higher

1. Address the underlying storage connectivity issues before proceeding with Manager appliance specific steps.
2. Reboot the manager node (fsck will run at boot time on all partitions)
3. If corfu partitions (/config or /nonconfig) get mounted as read-only and a reboot does not help, follow the instructions in NSX Manager File system is corrupted or goes into read only mode.
4. If a reboot does not help and other a partition other than one of the corfu partitions noted in Step 3 is mounted as read-only, use the following steps to resolve the issue.

Notes:

• • If possible, take a backup before proceeding.
  • Preforming the following file system repair steps will only fix the OS file system and not repair any corruption which may have occurred to the NSX components. Any corruption may lead to other issues in the future.

1. 1. Connect to the console of the manager appliance
   2. Reboot the system
   3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly.
      Note: If you wait too long and the boot sequence does not pause, you must reboot the system again
   4. Keep the cursor on the Ubuntu selection
   5. Press e to edit the selected option
   6. Enter the user name (root) and the GRUB password for root (not the same as the appliance user root).
      Note: The default password is VMware1 before release 3.2 and NSX@VM!WaR10 for 3.2 and beyond.
   7. Search for the line starting with linux
   8. Remove all options after root= and add rw single init=/bin/bash after the UUID
   9. Press Ctrl-X to boot
   10. When the log messages stop, press Enter.
       Note: You will see the prompt root@(none):/#
   11. Run following commands to repair the file systems
       
       Non-corfu partitions:
       

       e2fsck -y /dev/sda1
       e2fsck -y /dev/sda2
       e2fsck -y /dev/sda3
       e2fsck -y /dev/mapper/nsx-image
       e2fsck -y /dev/mapper/nsx-var+log
       e2fsck -y /dev/mapper/nsx-repository
       e2fsck -y /dev/mapper/nsx-tmp
       e2fsck -y /dev/mapper/nsx-var+dump

       Corfu partitions
       

       e2fsck -y /dev/mapper/nsx-config
       e2fsck -y /dev/mapper/nsx-config__bak
       e2fsck -y /dev/mapper/nsx-secondary

Additional Information

If you need to open a Broadcom support request, be sure to note all troubleshooting steps already performed and the result(s) of each.