Manager node disk/partition is mounted as read-only alarm in NSX Manager Disk Corruption correction using FSCK

Products

• VMware NSX

Issue/Introduction

Title: Alarm for Manager disk/partition is mounted as read-only
Event ID: manager_health.storage_error

Alarm Description

• Purpose: Indicates whether a Manager disk/partition is mounted as read-only.
• Impact: Some functionality will not work if any of the manager partitions are mounted as read-only (e.g., login errors, disk write errors).

Environment

VMware NSX

VMware NSX-T Data Center

Cause

It may be because of the following reasons:

• File System Corruption: When the file system is corrupted, the Manager appliance OS might mount it as read-only to prevent further damage.
• Improper System Shutdown: Abrupt power loss or an unclean system shutdown can lead to a read-only file system.
• Hardware Issues: Faulty hard drives or damaged sectors can cause the file system to switch to read-only mode.
• Incorrect File System Configuration: Mistakes in the /etc/fstab file or other configuration files can result in a read-only file system.

Resolution

Steps to Resolve

For VMware NSX-T 3.0.2 and higher

1. Address the underlying storage connectivity issues, before proceeding with the below Manager appliance specific steps.
2. Reboot the manager node (fsck will run at boot time on all partitions).
3. If the corfu partitions (/config or /nonconfig) get mounted as read-only and a reboot does not help:
   1. The partitions /config and /nonconfig are used by the NSX-T database corfu; Linux recovery tools such as fsck may not repair these partitions.
   2. In such cases, where a reboot has not repaired these two partitions, it is advised to restore the appliances from backup to a point before the outage occurred.
   3. For details on restoring from backup, please review the VMware NSX Administration Guide.

Note: If only a single appliance was impacted, then you can remove that appliance and deploy a new one.

4. If a reboot does not help and another partition, other than the ones used for corfu partitions noted in Step 3, are mounted in read-only, use the following steps to resolve the issue.

Notes:

• If possible, take a backup before proceeding.
• Performing the following file system repair steps will only fix the OS file system and not repair any corruption which may have occurred to the NSX components.
• Any corruption may lead to other issues in the future.

1. Connect to the console of the manager appliance.
2. Reboot the system.
3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly.
   Note: If you wait too long and the boot sequence does not pause, you must reboot the system again.
4. Press e to edit the selected option 'Ubuntu' (Not Advanced Ubuntu).
5. Enter the user name (root) and the GRUB password for root (not the same as the appliance user root).
   Note: The default password is VMware1 before release 3.2 and NSX@VM!WaR10 for 3.2 and beyond.
6. Search for the line starting with linux.
7. At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 to boot with these options. This should bring up the prompt to login with the appliance root user credentials. If not, continue to step 10.

Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).

10. Remove all options after root= and add rw init=/bin/bash after the UUID:
    
11. Press Ctrl-X to boot.
12. When the log messages stop, press Enter.
    Note: You will see the prompt root@(none):/#
13. Run following commands to repair the file systems:
    Non-corfu partitions:

    e2fsck -y /dev/sda1
    e2fsck -y /dev/sda2
    e2fsck -y /dev/sda3
    e2fsck -y /dev/mapper/nsx-image
    e2fsck -y /dev/mapper/nsx-var+log
    e2fsck -y /dev/mapper/nsx-repository
    e2fsck -y /dev/mapper/nsx-tmp
    e2fsck -y /dev/mapper/nsx-var+dump

    Do NOT run e2fsck against CORFU partitions.

Additional Information

If you need to open a Broadcom support request, be sure to note all troubleshooting steps already performed and the result(s) of each and confirm the underlying storage issue is resolved.