HCX booting up in Emergency Mode after ESXi Host where the appliance was running sustained prolonged power outage

Products

• VMware HCX

Issue/Introduction

• vCenter server is not reporting IP address of the appliance
• HCX Manager UI is inaccessible
• HCX Manager https://<HCX_Manager>9443 page is inaccessible
• HCX Manager will only boot to emergency mode even after a reboot
• While viewing HCX Manager console window from vCenter, console window is indicating the HCX Manager is entering emergency mode

• No HCX Manager backups available

Cause

Unexpected appliance shutdown and appliance storage becoming unavailable

Resolution

• Boot HCX appliance to GRUB menu
  • Immediately press the e key when the Photon OS splash screen or GRUB menu appears to enter the GNU GRUB Edit Menu.
    • If necessary, Edit the HCX appliance boot options in VM options to delay boot time:  
      • https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-virtual-machine-administration-guide-8-0/configuring-virtual-machine-optionsvsphere-vm-admin/configuring-virtual-machine-boot-optionsvsphere-vm-admin/change-vm-boot-optionsvsphere-vm-admin.html
  • Find the line that starts with the word:  linux.
  • Go to the end of that line and add a space followed by: fsck.repair=yes.  If fsck.repair=yes does not work, can add: init=/bin/sh
    • When using: init=/bin/sh it is neccessary to run fsck -y manually from the shell prompt
  • Press ctrl-c or F10 to continue booting the appliance.
  • Verify all HCX services are in a running state:
    • systemctl status postgresdb
    • systemctl status zookeeper
    • systemctl status kafka
    • systemctl status app-engine
    • systemctl status web-engine
    • systemctl status appliance-managment

 

• After regaining access to the HCX UI, it may be necessary to redeploy Fleet Appliances (IX, WO, and NE) 
•  

Additional Information

For additional HCX service Troubleshooting steps please referrence the following KB articles

• Consecutive HCX Manager reboots may result in the app-engine and Kafka messaging services not coming online
• Failed to start HCX "appliance-management" service
• HCX Connector or Cloud Manager unresponsive due to high utilization of "/common" directory
• Recover HCX Manager and Fleet Appliances after datastore failure