Recover NSX manager if it fails to boot up with error "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY."
search cancel

Recover NSX manager if it fails to boot up with error "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY."

book

Article ID: 320303

calendar_today

Updated On:

Products

VMware NSX Networking

Issue/Introduction

The purpose of this KB is to provide steps to recover NSX manager node.

Symptoms:
NSX manager may report inconsistency in filesystem and fail to boot up. This has been observed post issues with storage infrastructure.
Following errors may be observed while rebooting manager node-


/dev/sda3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
fsck exited with status code 4
The root filesystem on /dev/sda3 requires a manual fsck
Rebooting automatically due to panic= boot argument

▶

Environment

VMware NSX-T Data Center

Resolution

Steps-

  1. Connect to the console of the appliance.
  2. Reboot the system.
  3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
  4. Press e to edit the menu.
  5. Choose the top Ubuntu line then enter the user name root and the GRUB password for root (not the same as the appliance's user root). The default password is VMware1 or NSX@VM!WaR10 depending upon version.
  6. Press e to edit the selected option.
  7. Search for the line starting with linux and delete everything to the right of "ro" and replace "ro" with  "rw" at the end.
  8. Edit this line to:


    Replace ro with rw


    Press Ctrl-X to boot.
  9. Run fsck on affected filesystem for e.g. fsck /dev/sda3.

 

After this the manager may show Login incorrect multiples times and fail to boot up properly. To recover from this do the following-

  1. Connect to the console of the appliance.
  2. Reboot the system.
  3. When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
  4. Press e to edit the menu.
  5. Choose the top Ubuntu line then enter the user name root and the GRUB password for root (not the same as the appliance's user root). The default password is VMware1.
  6. Press e to edit the selected option.
  7. Search for the line starting with linux and delete everything to the right of "ro" and replace "ro" with  "rw" at the end.
  8. Press Ctrl-X to boot.
  9. The appliance should boot to maintenance mode and you would be prompted to enter root password.
  10. Run the fsck /dev/mapper/nsx-tmp command. Press "Yes" to fix all errors encountered by the file system.
  11. Enter "Reboot" command, it should now boot into default mode.

 

Note: If you see boot issues even after running the above steps, run it again until Step 6 and on Step 7 add  'rw single init=/bin/bash' at the end of the line and run below commands on the next prompt.

e2fsck -y /dev/sda1
e2fsck -y /dev/sda2
e2fsck -y /dev/sda3
e2fsck -y /dev/mapper/nsx-config
e2fsck -y /dev/mapper/nsx-image
e2fsck -y /dev/mapper/nsx-var+log
e2fsck -y /dev/mapper/nsx-secondary
e2fsck -y /dev/mapper/nsx-config__bak

▶
Powered by Wolken Software