Manager node disk/partition is mounted as read-only alarm in NSX Manager - Disk Corruption correction using FSCK
search cancel

Manager node disk/partition is mounted as read-only alarm in NSX Manager - Disk Corruption correction using FSCK

book

Article ID: 330478

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Title: Alarm for Manager disk/partition is mounted as read-only
Event ID: manager_health.storage_error

Alarm Description:

  • Purpose: Indicates whether a Manager disk/partition is mounted as read-only.
  • Impact: Some functionality will not work if any of the manager partitions are mounted as read-only (e.g., login errors, disk write errors).

Symptoms observed:

  • In the /var/log/syslog, you may see log messages similar to the log lines below.
    2023-06-30T01:34:55.506234+00:00 nsxtmgr.vcf kernel - - - [6869346.074509] sd 2:0:1:0: [sdb] tag#1 CDB: Write(10) 2a 00 04 ## ## ## 00 02 78 00 2023-06-30T01:34:55.506238+00:00 nsxtmgr.vcf kernel - - - [6869346.074512] print_req_error: 1 callbacks suppressed 2023-06-30T01:34:55.506240+00:00 nsxtmgr.vcf kernel - - - [6869346.074516] print_req_error: I/O error, dev sdb, sector 78634720 2023-06-30T01:34:55.513497+00:00 nsxtmgr.vcf kernel - - - [6869346.075123] EXT4-fs warning: 3 callbacks suppressed 2023-06-30T01:34:55.513521+00:00 nsxtmgr.vcf kernel - - - [6869346.075127] EXT4-fs warning (device dm-8): ext4_end_bio:323: I/O error 10 writing to inode 4194321 (offset 85286912 size 872448 starting block 9828828)
  • Filesystem goes into read only mode nvpapi/api_server.log , you may see log messages similar to the log lines below.
    2023-07-13T03:11:02.145Z napi.root.node.services.utils.node-mgmt ERROR File system(s) found to be in read-only mode: /dev/sdb1 -> /mnt
  • "mount" command show file system used by appliance in a readonly mode.
  • Any of the filesystem used by appliance if full.
  • Not able to login into appliance or repetitive login failure message.

Environment

VMware NSX

VMware NSX-T Data Center

Cause

It may be because of the following reasons:

  • File System Corruption: When the file system is corrupted, the Manager appliance OS might mount it as read-only to prevent further damage.
  • Improper System Shutdown: Abrupt power loss or an unclean system shutdown can lead to a read-only file system.
  • Hardware Issues: Faulty hard drives or damaged sectors can cause the file system to switch to read-only mode.
  • Incorrect File System Configuration: Mistakes in the /etc/fstab file or other configuration files can result in a read-only file system.

Resolution

Steps to Resolve

For VMware NSX-T 3.0.2 and higher

  1. Address the underlying storage connectivity issues, before proceeding with the below Manager appliance specific steps.

  2. Reboot the manager node (fsck will run at boot time on all partitions).

  3. If the corfu partitions (/config or /nonconfig) get mounted as read-only and a reboot does not help:
    1. The partitions /config and /nonconfig are used by the NSX-T database corfu; Linux recovery tools such as fsck may not repair these partitions.
    2. In such cases, where a reboot has not repaired these two partitions, it is advised to restore the appliances from backup to a point before the outage occurred.
    3. For details on restoring from backup, please review the VMware NSX Administration Guide.
Note: If only a single appliance was impacted, then you can remove that appliance and deploy a new one.
  1. If a reboot does not help and another partition, other than the ones used for corfu partitions noted in Step 3, are mounted in read-only, use the following steps to resolve the issue.
Notes:
  • If possible, take a backup before proceeding.
  • Performing the following file system repair steps will only fix the OS file system and not repair any corruption which may have occurred to the NSX components.
  • Any corruption may lead to other issues in the future.
  1. Connect to the console of the manager appliance.

  2. Reboot the system.

  3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly.
    Note: If the GRUB menu does not appear, the appliance must be redeployed.
    Note:     If you wait too long and the boot sequence does not pause, you must reboot the system again.

  4. Press e to edit the selected option 'Ubuntu' (Not Advanced Ubuntu).

  5. Enter the user name (root) and the GRUB password for root (not the same as the appliance user root).

    Note: The default password is VMware1 before release 3.2 and NSX@VM!WaR10 for 3.2 and beyond.

  6. Search for the line starting with linux.

  7. At the end of this line, add fsck.mode=force fsck.repair=yes and press F10 to boot with these options. This should bring up the prompt to login with the appliance root user credentials. If not, continue to step h.

    Note: It is strongly recommended to redeploy new Manager nodes after recovering them, to replace the affected node(s).

  8. Remove all options after root= and add rw init=/bin/bash after the UUID:
  9. Press Ctrl-X to boot.

  10. When the log messages stop, press Enter.
    Note: You will see the prompt root@(none):/#

  11. Run following commands to repair the file systems:
    Non-corfu partitions:
    e2fsck -y /dev/sda1
e2fsck -y /dev/sda2
e2fsck -y /dev/sda3
e2fsck -y /dev/mapper/nsx-image
e2fsck -y /dev/mapper/nsx-var+log
e2fsck -y /dev/mapper/nsx-repository
e2fsck -y /dev/mapper/nsx-tmp
e2fsck -y /dev/mapper/nsx-var+dump
    Do NOT run e2fsck against CORFU partitions.

Additional Information

If you need to open a Broadcom support request, be sure to note all troubleshooting steps already performed and the result(s) of each and confirm the underlying storage issue is resolved.

Powered by Wolken Software