NSX Edge file system becomes Read-only

Products

VMware NSX for vSphere

Issue/Introduction

In the vsm.log file on NSX Manager, you see entries similar to:

INFO SimpleAsyncTaskExecutor-1 StatusAndStatsUtil:879 - Propogating vse event EDGE_FS_READONLY Module: vShield Edge Appliance Severity Critical

INFO SimpleAsyncTaskExecutor-1 EdgeUtils:315 - SysEvent-Detailed-Message :(Kept only in logs) :: File system is read only.

INFO SimpleAsyncTaskExecutor-1 EdgeUtils:316 - populateSystemEvent parameters : sourceName edge-#, morefIdOfObjectOnVc vm-###, moduleName vShield Edge Appliance, eventCode EDGE_FS_READONLY, severity Critical, messageParams [edge-#, <Edge-Name>-0] eventMetaData {edgeId=edge-#, edgeVmName=<Edge-Name>-0, edgeVmId=vm-###}

INFO SimpleAsyncTaskExecutor-1 SystemEventDaoImpl:133 - [SystemEvent] Time:'Mon Sep 12 21:56:40.000 GMT 2016', Severity:'Critical', Event Source:'edge-#', Code:'30181', Event Message:'NSX Edge edge-# VM name <Edge-Name>-0 file system is read only.', Module:'vShield Edge Appliance', Universal Object:'false'

There are significant issues in the vSphere environment, you may see entries similar to:

[APDCorrelator] 522006000566us: [vob.storage.apd.start] Device or filesystem with identifier [########-####31b3] has entered the All Paths Down state

Environment

VMware NSX for vSphere

Cause

Underlying storage issues such as disconnect from storage array, disk drive failures, network interruption etc. can cause an EDGE to enter into a READ-ONLY mode to protect its data integrity from further impact. This is an expected behavior from any Linux based file system.

Related information: Linux based file systems become read-only

Resolution

To resolve the issue, one must first address the underlying storage connectivity issues and confirm that underlying storage is healthy and optimal before proceeding with any of these steps.

NOTE: DO NOT PROCEED IF UNDERLYING STORAGE IS DEGRADED

Related Information:

Typically if an EDGE enters into READ-ONLY Mode, SSH access to the EDGE may not be available and the only way to login would be via the CONSOLE SCREEN. Below steps will need to be followed for the recovery:

If not already in Emergency Mode on the CONSOLE with SHELL Access, rebooting the EDGE will either clear out the bad i-node table or will drop you into an emergency mode.
Press Control-D to provide Root Access and SHELL.
From the SHELL type 'journalctl -xb' to search for the word "failed"
We are interested to find all the File System Check Failure on any /dev/mapper locations. These are generally mount point locations. For example, /dev/mapper/nsx-var+log . Write all of these /dev/mapper location down, as we will have to go through each one of them individually
From the SHELL type 'e2fsck -y [path of the mount] ' e.g e2fsck -y /dev/mapper/nsx-var_log'
Repeat the step for each of the failed path found previously in the journalctl -xb
Once Complete Reboot the EDGE again to confirm integrity of file system. If you return into emergency mode again contact VMware by Broadcom Support.

Alternately, below steps can be followed for the recovery of the node:

Re-deploy the edge.
Reboot the edge (Both edges in HA pair).
Force sync the edge.

Additional Information

Please refer to NSX-T edge file system becomes Read-Only for NSX-T Edge.