After upgrading to ESXi 8.0.x, host UI unavailable and says "HTTP 503 - no healthy upstream"

Products

• VMware vSphere ESXi
• VMware vSphere ESX 8.x

Issue/Introduction

Post-upgrade of ESXi host,

• Hosts within the cluster may enter a disconnected state in vCenter.
• Accompanied by persistent crashing of the hostd service, even after manual attempts to restart the hostd service.
• You may see errors similar to below entries in hostd service log :
  
  /var/run/log/hostd.log

Hostd[<TID>]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort ## on dvs ## ## ## ## ## ## ## not found.
Hostd[<TID>]: [Originator@6876 sub=Libs] info [ConfigStore:3e98e02e00] [cs:1:3282982XX]CancelTransaction invoked.
Hostd[<TID>]: [Originator@6876 sub=Default] error: N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound
Hostd[<TID>]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort ## on dvs ## ## ## ## ## ## ##  not found
Hostd[<TID>]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort ## on dvs ## ## ## ## ## ## ## not found
Hostd[<TID>]: error -[2109312] [Originator@6876 sub=VMOMI-FSS] Unknown feature 'VMODL_LIFECYCLE'. ASSUMING DISABLED
Hostd[<TID>]: [Originator@6876 sub=Libs] error [ConfigStore:2d49e81e00] [1046] Not a non-singleton object:  key browsable_console_dir
Hostd[<TID>]: [Originator@6876 sub=Libs] info [ConfigStore:2d49e81e00] ConfigStoreException: [context]zKq7AVICAgAAAP////8LY29uZmlnc3RvcmUAANPSAmxpYmNvbmZpZ3N0b3JlLnNvAAAI1AMAdlwDAaZwAmxpYnZpbWNvbmZpZ3N0b3JlLnNvAAHDwwMBWF8BAmBCXWhvc3RkAAKJeF0CGhdSAx00AmxpYmMuc28uNgACNdhT[/context]
Hostd[<TID>]: [Originator@6876 sub=Libs] error [ConfigStore:2d49e81e00] [1046] Not a non-singleton object:  key default_config_path
Hostd[<TID>]: [Originator@6876 sub=Libs] info [ConfigStore:2d49e81e00] ConfigStoreException: [context]zKq7AVICAgAAAP////8LY29uZmlnc3RvcmUAANPSAmxpYmNvbmZpZ3N0b3JlLnNvAAAI1AMAdlwDAaZwAmxpYnZpbWNvbmZpZ3N0b3JlLnNvAAHDwwMBWF8BAmBCXWhvc3RkAAKJeF0CGhdSAx00AmxpYmMuc28uNgACNdhT[/context]

Cause

• The hostd service fails to start after an upgrade due to a DVS configuration mismatch.
• This occurs when the required conversion of network settings to a new format fails because the pre-upgrade configuration was already incorrect.

Resolution

Follow below steps to identify and fix the invalid configuration in ESXi Config Store. These steps can be also followed to confirm if there is a mismatch before performing the ESXi upgrade :

1. SSH to the ESXi host
2. Execute the below command to list the vSwitch details:
   
   esxcfg-vswitch -l
   
   You should get output like:
   
   DVS Name             Num Ports      Used Ports        Configured Ports           MTU       Uplinks   
DvS_xxxx                9216            9                     512               1600     vmnic1,vmnic0

  DVPort ID                            In Use              Client      
  96                                      0           
  97                                      0           
  5                                       1                 vmk0        
  122                                     1                 vmk1        
  181                                     1                 vmnic1      
  180                                     1                 vmnic0   

 

3. Execute the below command to list the vSwitch details in Config Store:
   
configstorecli config current get -c esx -g network_vds -k switch_solutions

[
   {
      "configured_max_ports": 512,
      "lags": [
         {
            "lag_id": "xxxxxx"
         }
      ],
      "name": "DvS_xxxx",
      "portset_name": "DvsPortset-xx",
      "product": {
         "forwarding_class": "cswitch"
      },
      "uplinks": [
         {
            "connection_cookie": 0,
            "nic": "vmnic3",
            "uplink_port_key": "99"
         },
         {
            "connection_cookie": 0,
            "nic": "vmnic2",
            "uplink_port_key": "98"
         },
         {
            "connection_cookie": 0,
            "nic": "vmnic0",
            "uplink_port_key": "180"
         },
         {
            "connection_cookie": 0,
            "nic": "vmnic1",
            "uplink_port_key": "181"
         }
      ],
      "uuid": "xxxxxxxxxxxxxxxxxxxx"
   }
]

   
   
4. Check if all the uplink_port_key in the second command are shown in the first command(DVPort ID).
   1. If Yes, then there is no mismatch in the ESXi Config Store.
   2. If No, run the attached Python script (CorrectUplinkPortKey.py) by modifying the fields (VDS_NAME and PORT_KEY_MAP)
      
      Note: Since executing the script modifies the ConfigStore on a specific ESXi host, it is highly recommended to place the ESXi host into Maintenance Mode to avoid any potential impact on production workloads. If assistance is needed, contact Broadcom Technical Support.

Example:
VDS_NAME = "DvS_XXXX"
PORT_KEY_MAP = [("98","96"), ("99","97")]             Note:(98 and 99 are mapped with unused ports in the first command output)

1. 1. 1. Backup the /etc/vmware/configstore/current-store-1
      2. Assign execution permission to the script and execute it:
         
         python3 CorrectUplinkPortKey.py
         
         
      3. Upon execution of the script, you should see the below messages:
         
         Update port key from 98 to 96
Update port key from 99 to 97
Successfully updated the port keys

 

5. Check the VDS configuration again to confirm there is no mismatch in Config store.
   
configstorecli config current get -c esx -g network_vds -k switch_solutions
   
   You should find that "99" is changed to "97" and "98" is changed to "96" in above example.
   
   
6. Start hostd service:
   
   /etc/init.d/hostd start
   
   
7. After a few seconds, the host should be shown as connected in VC.

Additional Information

Please refer to the KB "Hostd Service Crash on ESXi After Upgrade from 7.0 U3 to 8.0 U3" if stale entries are found in the config store