Manager node disk/partition is mounted as read-only alarm in NSX Manager

Products

VMware NSX

Issue/Introduction

Title: Alarm for Manager disk/partition is mounted as read-only

Event ID: `manager_health.storage_error`

Alarm Description:

Purpose: Indicates whether a Manager disk/partition is mounted as read-only.
Impact: Some functionality will not work if any of the manager partitions are mounted as read-only (e.g., login errors, disk write errors).

======

In the /var/log/syslog, you may see log messages similar to the log lines below:

YYYY-MM-DDT01:34:55.506234+00:00 nsxtmgr.vcf kernel - - - [6869346.074509] sd 2:0:1:0: [sdb] tag#1 CDB: Write(10) 2a 00 04 ## ## ## 00 02 78 00 YYYY-MM-DDT01:34:55.506238+00:00 nsxtmgr.vcf kernel - - - [6869346.074512] print_req_error: 1 callbacks suppressed YYYY-MM-DDT01:34:55.506240+00:00 nsxtmgr.vcf kernel - - - [6869346.074516] print_req_error: I/O error, dev sdb, sector 78634720 YYYY-MM-DDT01:34:55.513497+00:00 nsxtmgr.vcf kernel - - - [6869346.075123] EXT4-fs warning: 3 callbacks suppressed YYYY-MM-DDT01:34:55.513521+00:00 nsxtmgr.vcf kernel - - - [6869346.075127] EXT4-fs warning (device dm-8): ext4_end_bio:323: I/O error 10 writing to inode 4194321 (offset 85286912 size 872448 starting block 9828828)

In nvpapi/api_server.log , you may see log messages similar to the log lines below indicating which file systems are in read-only mode:

YYYY-MM-DDT03:11:02.145Z napi.root.node.services.utils.node-mgmt ERROR File system(s) found to be in read-only mode: /dev/sdb1 -> /mnt

The "mount" command shows file systems used by NSX Manager in a readonly mode.
Not able to login into appliance or repetitive login failure message.
NSX Manager appliance console may not proceed past a line stating: kauditd hold queue overflow

Environment

VMware NSX

VMware NSX-T Data Center

Cause

The following are common causes of NSX Manager nodes' partitions going into a read-only state:

File System Corruption: When the file system is corrupted, the Manager appliance OS might mount it as read-only to prevent further damage.
Improper System Shutdown: Abrupt power loss or an unclean system shutdown can lead to a read-only file system.
Hardware Issues: Faulty hard drives or damaged sectors can cause the file system to switch to read-only mode.
Incorrect File System Configuration: Mistakes in the /etc/fstab file or other configuration files can result in a read-only file system.

Resolution

Steps to Resolve

For VMware NSX-T 3.0.2 and higher

Address any underlying storage connectivity issues before proceeding with the below Manager appliance specific steps.
Reboot the manager node. Filesystem checks (fsck) should run at boot time on all partitions.
If the Corfu partitions (/config or /nonconfig) get mounted as read-only and a reboot does not help:
1. The partitions /config and /nonconfig are used by the NSX database Corfu. Linux recovery tools such as fsck may not repair these partitions.
2. In such cases, where a reboot has not repaired these two partitions, it is advised to restore the appliances from backup to a point before the outage occurred.
3. For details on restoring from backup, review the VMware NSX Administration Guide.
  
  Note: If only a single appliance was impacted, you can remove that single appliance and deploy a new one.

If a reboot does not help, and another partition other than /config and /nonconfig are mounted in read-only, use the steps below to resolve the issue.

Notes:

If possible, take a backup before proceeding.
Performing the following file system repair steps will only fix the OS file system and not repair any corruption which may have occurred to the NSX components.
Any corruption may lead to other issues in the future.

1. Connect to the console of the manager appliance.
2. Reboot the system.
3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly.
  Note: If the GRUB menu does not appear, the appliance must be redeployed.
  Note: If you wait too long and the boot sequence does not pause, you must reboot the system again.
4. Press e to edit the selected option 'Ubuntu' (Not Advanced Ubuntu).
5. Enter the user name (root) and the GRUB password (not the same as the appliance user root).
  
  Note: The default password is VMware1 before release 3.2 and NSX@VM!WaR10 for 3.2 and beyond.
6. Search for the line starting with linux.
7. At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 to boot with these options. This should bring up the prompt to login with the appliance root user credentials. If not, continue to step h.
  
  Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).
8. Remove all options after root= and add rw init=/bin/bash after the UUID:
9. Press Ctrl-X to boot.
10. When the log messages stop, press Enter.
  Note: You will see the prompt root@(none):/#
11. Run the following commands to repair the non-Corfu file systems:
```
e2fsck -y /dev/sda1
e2fsck -y /dev/sda2
e2fsck -y /dev/sda3
e2fsck -y /dev/mapper/nsx-image
e2fsck -y /dev/mapper/nsx-var+log
e2fsck -y /dev/mapper/nsx-repository
e2fsck -y /dev/mapper/nsx-tmp
e2fsck -y /dev/mapper/nsx-var+dump
```
  Do NOT run e2fsck against CORFU partitions.

Additional Information

If you need to open a Broadcom support request, be sure to note all troubleshooting steps already performed and the result(s) of each and confirm the underlying storage issue is resolved.

For more information, see Creating and managing Broadcom support cases.