NSX Manager File system is corrupted or goes into read only mode.

Products

• VMware NSX

Issue/Introduction

Symptoms:

• 1. In the /var/log/syslog, you may see log messages similar to the log lines below.

2023-06-30T01:34:55.506234+00:00 nos-wld-nsxtmn02.vcf.netone.local kernel - - - [6869346.074509] sd 2:0:1:0: [sdb] tag#1 CDB: Write(10) 2a 00 04 af de e0 00 02 78 00
2023-06-30T01:34:55.506238+00:00 nos-wld-nsxtmn02.vcf.netone.local kernel - - - [6869346.074512] print_req_error: 1 callbacks suppressed
2023-06-30T01:34:55.506240+00:00 nos-wld-nsxtmn02.vcf.netone.local kernel - - - [6869346.074516] print_req_error: I/O error, dev sdb, sector 78634720
2023-06-30T01:34:55.513497+00:00 nos-wld-nsxtmn02.vcf.netone.local kernel - - - [6869346.075123] EXT4-fs warning: 3 callbacks suppressed
2023-06-30T01:34:55.513521+00:00 nos-wld-nsxtmn02.vcf.netone.local kernel - - - [6869346.075127] EXT4-fs warning (device dm-8): ext4_end_bio:323: I/O error 10 writing to inode 4194321 (offset 85286912 size 872448 starting block 9828828)

• 2. Filesystem goes into read only mode nvpapi/api_server.log , you may see log messages similar to the log lines below.

2023-07-13T03:11:02.145Z napi.root.node.services.utils.node-mgmt ERROR File system(s) found to be in read-only mode: /dev/sdb1 -> /mnt

• 3. "mount" command show file system used by appliance in a readonly mode.
• 4. Any of the filesystem used by appliance if full.
• 5. Not able to login into appliance or repetitive login failure message.

Environment

VMware NSX-T Data Center

Cause

It may cause because of following reason

• 1. File System Corruption: When the file system is corrupted, Manager appliance OS  might choose to mount it as read-only to prevent further damage.
• 2. Improper System Shutdown: Abrupt power loss or an unclean system shutdown can lead to a read-only file system.
• 3. Hardware Issues: Faulty hard drives or damaged sectors can cause the file system to switch to read-only mode.
• 4. Incorrect File System Configuration: Mistakes in the /etc/fstab file or other configuration files can result in a read-only file system.

Resolution

If there is a file system corruption, the OS level file system may be repaired using disk utilities, however, the VMware components such as corfu is not repaired using these tools.
In such cases where file system corruption may have occurred and the system is in read only mode, it is advised to restore the appliances from back, to a point before the outage occurred.
For details on restoring from backup, please review the VMware NSX Administration Guide.

Workaround:

• Please note, backups are a requirement in production environments.
• If you believe you have encountered this issue and are unable to restore from backup, please open a support request with Broadcom Support and refer to this KB article.