Manager node disk/partition is mounted as read-only alarm in NSX Manager Disk Corruption correction using FSCK
search cancel

Manager node disk/partition is mounted as read-only alarm in NSX Manager Disk Corruption correction using FSCK

book

Article ID: 330478

calendar_today

Updated On: 01-02-2025

Products

VMware NSX

Issue/Introduction

Title: Alarm for Manager disk/partition is mounted as read-only
Event ID: manager_health.storage_error

Alarm Description

  • Purpose: Indicates whether a Manager disk/partition is mounted as read-only
  • Impact: Some functionality will not work if any of the manager partition gets mounted as read-only (like - login error, disk write error)

Environment

VMware NSX-T Data Center

Cause

It may be because of following reasons:

  • File System Corruption: When the file system is corrupted, Manager appliance OS  might choose to mount it as read-only to prevent further damage.
  • Improper System Shutdown: Abrupt power loss or an unclean system shutdown can lead to a read-only file system.
  • Hardware Issues: Faulty hard drives or damaged sectors can cause the file system to switch to read-only mode.
  • Incorrect File System Configuration: Mistakes in the /etc/fstab file or other configuration files can result in a read-only file system.

Resolution

Steps to Resolve
For VMware NSX-T 3.0.2 and higher

  1. Address the underlying storage connectivity issues, before proceeding with the below Manager appliance specific steps.
  2. Reboot the manager node (fsck will run at boot time on all partitions).
  3. If the corfu partitions (/config or /nonconfig) get mounted as read-only and a reboot does not help.
    1. The partitions /config and /nonconfig are used by the NSX-T database corfu, Linux recovery tools such as fsck may not repair these partitions.
    2. In such cases, where a reboot has not repaired these two partitions, it is advised to restore the appliances from backup, to a point before the outage occurred.
    3. For details on restoring from backup, please review the VMware NSX Administration Guide.
      Note: If only a single appliance was impacted, then you can remove that appliance and deploy a new one.
  4. If a reboot does not help and another partition, other than the one's used for corfu partitions noted in Step 3, are mounted in read-only, use the following steps to resolve the issue.

Notes:

    • If possible, take a backup before proceeding.
    • Performing the following file system repair steps will only fix the OS file system and not repair any corruption which may have occurred to the NSX components.
    • Any corruption may lead to other issues in the future.
    1. Connect to the console of the manager appliance
    2. Reboot the system
    3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly.
      Note: If you wait too long and the boot sequence does not pause, you must reboot the system again
    4. Keep the cursor on the Ubuntu selection
    5. Press e to edit the selected option
    6. Enter the user name (root) and the GRUB password for root (not the same as the appliance user root).
      Note: The default password is VMware1 before release 3.2 and NSX@VM!WaR10 for 3.2 and beyond.
    7. Search for the line starting with linux
    8. At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 to boot with these options. This should bring up the prompt to login with the appliance root user credentials. If not, continue to step J.
    9. Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).  
    10. Remove all options after root= and add rw single init=/bin/bash (OR rw init=/bin/bash ) after the UUID:

    11. Press Ctrl-X to boot
    12. When the log messages stop, press Enter.
      Note: You will see the prompt root@(none):/#
    13. Run following commands to repair the file systems

      Non-corfu partitions:
      e2fsck -y /dev/sda1
      e2fsck -y /dev/sda2
      e2fsck -y /dev/sda3
      e2fsck -y /dev/mapper/nsx-image
      e2fsck -y /dev/mapper/nsx-var+log
      e2fsck -y /dev/mapper/nsx-repository
      e2fsck -y /dev/mapper/nsx-tmp
      e2fsck -y /dev/mapper/nsx-var+dump
      Corfu partitions
      e2fsck -y /dev/mapper/nsx-config
      e2fsck -y /dev/mapper/nsx-config__bak
      e2fsck -y /dev/mapper/nsx-secondary

Additional Information

If you need to open a Broadcom support request, be sure to note all troubleshooting steps already performed and the result(s) of each and confirm the underlying storage issue is resolved.