Recover Avi LB controllers and service engine VMs from Read-Only partitions

Products

• VMware Avi Load Balancer

Issue/Introduction

Symptoms:
- Nodes with read-only filesystem will go into an inactive state. 
- Cluster will not be converged/stable.
- Multiple processes on the controller will fail to come up.
- SE will be marked down on Avi. 

Environment

VMware NSX Advanced Load Balancer

Resolution

- Launch the console of the VM that is affected.
- If the VM has failed to boot up, you should see an error like "The root filesystem on /dev/sda3 requires a manual fsck"



- Please note that the root filesystem could be mounted on a different path than "/dev/sda3". Make a note of the error and it will highlight the path that needs to be repaired.
- Once you have access to the console, run the following command:

fsck.ext4 -f /dev/sda3

- The path after -f will be the path mentioned in the error.
- After the command has been run, you should see that the file system has been modified:


- After this has been run, reboot the VM and this time it should boot up successfully.
- If the issue is not resolved after these steps, please open a case with technical support.

- This issue is caused due to an underlying storage issue and the above steps are just to recover the file-system for Avi VMs. 

Additional Information

Impact/Risks:
- When the disk goes into this state, the VMs are not properly booted up leading to multiple issues.
- This can affect the user's ability to access GUI and make changes and it can also affect the SE's ability to process traffic correctly.