Hostd Service Crash on ESXi After Upgrade from 7.0 U3 to 8.0 U3
search cancel

Hostd Service Crash on ESXi After Upgrade from 7.0 U3 to 8.0 U3

book

Article ID: 412202

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • An ESXi host was successfully upgraded to version 8.0 Update 3 using vSphere Lifecycle Manager. However, after the upgrade, the hostd service continuously crashes, causing the host to appear disconnected in the vSphere Client. Additionally, access to the Host Client UI fails with the error:
    HTTP 503 – No healthy upstream
  • Manual attempts to restart the hostd service are unsuccessful.
  • The /var/run/log/hostd.log file includes entries similar to the following:
    Hostd[2118553]: - time the service was last started ###-###-###, Section for VMware ESX, pid=#####, version=###, build=#######, option=Release
Hostd[2118553]: [Originator@6876 sub=Default] error: N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound
Hostd[2118553]: --> )
Cr(162) Hostd[2118553]: --> [context]zKq7AVICAgAAANVUewESaG9zdGQA...........................................Whvc3RkAAHSQmIBpj+LAe1CiwH+xpQB262WAW2ylgHRrm4Bi8FuAbp6bgCd00gBNRSaAaA2mgHqSpoBnFhZAj00AmxpYmMuc28uNgABGmxd[/context]

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

  • The hostd service was crashing due to unresolved references to physically removed NICs (vmnic6 and vmnic7) that remained in the host config store.
  • Although vmnic6 and vmnic7 were no longer present on the Distributed Virtual Switch (DVS), stale entries referencing them persisted in the host config store. These invalid references caused hostd to fail during initialization, resulting in the host appearing disconnected in the vSphere Client and the Host Client UI being unavailable.
  • configstorecli output showing stale NIC references:
    { "connection_cookie": 0, "nic": "vmnic6", "uplink_port_key": "184" },
{ "connection_cookie": 0, "nic": "vmnic7", "uplink_port_key": "185" }

Resolution

This article outlines steps to identify and resolve mismatches between the ESXi host’s active Distributed vSwitch (DVS) configuration and the host’s Config Store.

These steps are helpful:

  • Before upgrading ESXi, to prevent upgrade failures.

  • When troubleshooting hostd crashes caused by stale DVS configuration.


Prerequisites

  • SSH access to the ESXi host.

  • The host must be placed in maintenance mode.

  • Migrate all VMs off the host if configuration changes are needed.


Step 1: Check Active DVS Configuration

  1. SSH into the ESXi host.

  2. Run:
    esxcfg-vswitch -l

  3. Review the DVS section. Example output:

    DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
CTVDS-CTX        9216        21          512               1500    vmnic9,vmnic8,vmnic1,vmnic2,vmnic3,vmnic4,vmnic5,vmnic0

  DVPort ID      In Use      Client
  40             1           vmnic0
  41             1           vmnic1
  42             1           vmnic2
  43             1           vmnic3
  44             1           vmnic4
  45             1           vmnic5
  46             1           vmnic8
  47             1           vmnic9
  9              1           vmk1
  17             1           vmk2
  25             1           vmk3
  2              1           vmk0
     
     

Step 2: Check DVS Details in Config Store

  • Run the following to review DVS configuration from the host's config store:
    configstorecli config current get -c esx -g network_vds -k switches

  • Example output:
    "configured_max_ports": 512, "uplinks": [{"nic": "vmnic0", "uplink_port_key": "40", "connection_cookie": 0}, 
{"nic": "vmnic5", "uplink_port_key": "45", "connection_cookie": 0}, 
{"nic": "vmnic4", "uplink_port_key": "44", "connection_cookie": 0}, 
{"nic": "vmnic3", "uplink_port_key": "43", "connection_cookie": 0}, 
{"nic": "vmnic2", "uplink_port_key": "42", "connection_cookie": 0}, 
{"nic": "vmnic1", "uplink_port_key": "41", "connection_cookie": 0}, 
{"nic": "vmnic6", "uplink_port_key": "184", "connection_cookie": 0}, 
{"nic": "vmnic7", "uplink_port_key": "185", "connection_cookie": 0}, 
{"nic": "vmnic8", "uplink_port_key": "46", "connection_cookie": 0}, 
{"nic": "vmnic9", "uplink_port_key": "47", "connection_cookie": 0}], 
"product": {"forwarding_class": "cswitch"}}|{"portset_name": "DvsPortset-0", "name": "CTVDS-CTX"}|||3


Step 3: Validate Uplink Port Keys

  • Compare the uplink_port_key values from the host config store and output with the DVPort IDs from
    esxcfg-vswitch -l

  • If mismatched or stale vmnics exist, → Proceed with cleanup steps.


Step 4: Fix Mismatch in Config Store

   1. Prepare the Host

  •  Migrate all VMs off the host.

  • Put the host into Maintenance Mode.

  2. Export Current Configuration

  • For both vSphere 7.x and vSphere 8.x:
    configstorecli config current get -c esx -g network_vds -k switches -outfile /tmp/switch.json

  • Additional step for vSphere 8.x:
    configstorecli config current get -c esx -g network_vds -k switch_solutions -outfile /tmp/switch_solutions.json

  3. Back Up the Config Files
    cp /tmp/switch.json /var/core/switch.json cp /tmp/switch_solutions.json /var/core/switch_solutions.json


  4. Edit and Clean Up Stale VMNICs

  • Open the exported files using vi:
    vi /tmp/switch.json vi /tmp/switch_solutions.json

  • Remove entries referencing stale or non-existent vmnicX interfaces (e.g., vmnic6, vmnic7) that are no longer shown in
    esxcfg-vswitch -l.


Step 5: Import Cleaned Config

  • For vSphere 7.x:
    configstorecli config current set -c esx -g network_vds -k switches -infile /tmp/switch.json

  • For vSphere 8.x:
    configstorecli config current set -c esx -g network_vds -k switches -infile /tmp/switch.json
  • configstorecli config current set -c esx -g network_vds -k switch_solutions -infile /tmp/switch_solutions.json


Step 6: Restart hostd Service
            /etc/init.d/hostd restart


Step 7: Verify
Run the following commands to verify the removal of stale entries:

  • vSphere 7.x
    configstorecli config current get -c esx -g network_vds -k switches

  • vSphere 8.x
    configstorecli config current get -c esx -g network_vds -k switches
  • configstorecli config current get -c esx -g network_vds -k switch_solution

  • Also verify:
    • No stale vmnicX values are present.
    • Uplink port keys match
      esxcfg-vswitch -l

    • hostd is running:
      /etc/init.d/hostd status



Additional Information

Refer to the following knowledge base article if there's a DVPort ID mismatch between the DVS switch and the ESXi config store when vmnics are present.
After upgrading to ESXi 8.0.x, host UI unavailable and says "HTTP 503 - no healthy upstream"

Powered by Wolken Software