Manager node disk/partition is mounted as read-only alarm in NSX Manager Disk Corruption correction using FSCK
searchcancel
Manager node disk/partition is mounted as read-only alarm in NSX Manager Disk Corruption correction using FSCK
book
Article ID: 330478
calendar_today
Updated On: 01-02-2025
Products
VMware NSX
Issue/Introduction
Title: Alarm for Manager disk/partition is mounted as read-only Event ID: manager_health.storage_error
Alarm Description
Purpose: Indicates whether a Manager disk/partition is mounted as read-only
Impact: Some functionality will not work if any of the manager partition gets mounted as read-only (like - login error, disk write error)
Environment
VMware NSX-T Data Center
Cause
It may be because of following reasons:
File System Corruption: When the file system is corrupted, Manager appliance OS might choose to mount it as read-only to prevent further damage.
Improper System Shutdown: Abrupt power loss or an unclean system shutdown can lead to a read-only file system.
Hardware Issues: Faulty hard drives or damaged sectors can cause the file system to switch to read-only mode.
Incorrect File System Configuration: Mistakes in the /etc/fstab file or other configuration files can result in a read-only file system.
Resolution
Steps to Resolve For VMware NSX-T 3.0.2 and higher
Address the underlying storage connectivity issues, before proceeding with the below Manager appliance specific steps.
Reboot the manager node (fsck will run at boot time on all partitions).
If the corfu partitions (/config or /nonconfig) get mounted as read-only and a reboot does not help.
The partitions /config and /nonconfig are used by the NSX-T database corfu, Linux recovery tools such as fsck may not repair these partitions.
In such cases, where a reboot has not repaired these two partitions, it is advised to restore the appliances from backup, to a point before the outage occurred.
For details on restoring from backup, please review the VMware NSX Administration Guide. Note: If only a single appliance was impacted, then you can remove that appliance and deploy a new one.
If a reboot does not help and another partition, other than the one's used for corfu partitions noted in Step 3, are mounted in read-only, use the following steps to resolve the issue.
Notes:
If possible, take a backup before proceeding.
Performing the following file system repair steps will only fix the OS file system and not repair any corruption which may have occurred to the NSX components.
Any corruption may lead to other issues in the future.
Connect to the console of the manager appliance
Reboot the system
When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. Note: If you wait too long and the boot sequence does not pause, you must reboot the system again
Keep the cursor on the Ubuntu selection
Press e to edit the selected option
Enter the user name (root) and the GRUB password for root (not the same as the appliance user root). Note: The default password is VMware1 before release 3.2 and NSX@VM!WaR10 for 3.2 and beyond.
Search for the line starting with linux
At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 to boot with these options. This should bring up the prompt to login with the appliance root user credentials. If not, continue to step J.
Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).
Remove all options after root= and add rw single init=/bin/bash (OR rw init=/bin/bash ) after the UUID:
Press Ctrl-X to boot
When the log messages stop, press Enter. Note: You will see the prompt root@(none):/#
If you need to open a Broadcom support request, be sure to note all troubleshooting steps already performed and the result(s) of each and confirm the underlying storage issue is resolved.