Host becomes unresponsive after the upgrade from 8.0U1c to 8.0U2c
search cancel

Host becomes unresponsive after the upgrade from 8.0U1c to 8.0U2c

book

Article ID: 375658

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Host is unresponsive after upgrading from version 8.0U1c to 8.0U2c. We see that hostd keeps crashing without generating any core dumps.

hostd.log shows the below error

Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] GetSubObjectInstances : key uplinks
Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] GetSubObject : key uplinks id vmnic4
Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] GetSubObject: Could not find key: uplinks
Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] GetSubObject : key uplinks id vmnic2
Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] CreateSubObject : key uplinks id vmnic2
Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] GetSubObject : key uplinks id vmnic0
Hostd[2102376]: [Originator@6876 sub=Libs] trivia [ConfigStore:3e98e02e00] CreateSubObject : key uplinks id vmnic0
Hostd[2102376]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found

hostd.0:2024-08-12T14:44:45.065Z Db(167) Hostd[2100970]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found
hostd.0:2024-08-12T14:47:18.650Z Db(167) Hostd[2102090]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found
hostd.0:2024-08-12T14:47:19.216Z Db(167) Hostd[2102188]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found
hostd.0:2024-08-12T14:47:19.783Z Db(167) Hostd[2102232]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found
hostd.1:2024-08-12T14:39:53.314Z Db(167) Hostd[2101851]: [Originator@6876 sub=Hostsvc.NetworkProvider] SaveDVPortsWorker:DVS_NAME took 0ms, 00:00:00.000828 (hh:mm:ss.us).
hostd.log:2024-08-12T14:47:20.347Z Db(167) Hostd[2102276]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found
hostd.log:2024-08-12T14:47:20.917Z Db(167) Hostd[2102320]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found
hostd.log:2024-08-12T14:47:21.483Z Db(167) Hostd[2102376]: [Originator@6876 sub=Hostsvc.NetworkProvider] DVPort 85 on dvs 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1 not found

 

Environment

ESXi 8.0 Update 1c

Cause

The vDS actually has only 2 uplinks , but as per the logs , it was looking for a stale uplink (vmnic4). The Configstore

 

VCDB=# select * from vpx_dvs_uplink where id=1044712;
   id | uplink_port_name | uplink_index | surr_key | dpu_mode
---------+------------------+--------------+----------+----------
 1044712 | vmnic0_uplinks | 0 | 672421 |
 1044712 | vmnic2_uplinks | 1 | 672422 |
(2 rows)

 

The direct root cause is the following log that a DVPort 85 cannot be found on DVS 50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1. Hence, when booting, the DvsVmkProvider::GetHctlDVPort will throw the "NotFound" exception.

 

1) DVPort 85 was used by a staled vds uplink.

sqlite> select * from Config where ConfigGroup='network_vds';
    Component = esx
  ConfigGroup = network_vds
         Name = switches
  Identifier = DVS_NAME
 ModifiedTime = 2024-08-12 14:44:11
 CreationTime = 2024-08-12 14:44:09
      Version = 2.2
      Success = 1
AutoConfValue =
    UserValue = {"uplinks": [{"nic": "vmnic4", "uplink_name": "toBeUpgraded"},
                      {"nic": "vmnic2", "uplink_name": "toBeUpgraded"},
               {"nic": "vmnic0", "uplink_name": "toBeUpgraded"}],
        
         "name": "DVS_NAME"}
  VitalValue = {"portset_name": "DvsPortset-0", "name": "DVS_NAME"}
  CachedValue =
 DesiredValue =
     Revision = 3

    Component = esx
  ConfigGroup = network_vds
         Name = switch_solutions
  Identifier = DVS_NAME
 ModifiedTime = 2024-08-12 14:44:11
 CreationTime = 2024-08-12 14:44:09
      Version = 2.2
      Success = 1
AutoConfValue =
    UserValue = {"uuid": "50 26 0a 1a 44 76 df b4-79 53 d8 54 4c 8b c8 e1",
         "name": "DVS_NAME",
   "network_offload_spec_id": "None",
   "uplinks": [{"nic": "vmnic4", "connection_cookie": 0, "uplink_port_key": "85"},
               {"nic": "vmnic2", "uplink_port_key": "41", "connection_cookie": 0},
        {"nic": "vmnic0", "uplink_port_key": "40", "connection_cookie": 0}],
   "product": {"forwarding_class": "cswitch"},
     "configured_max_ports": 512}
  VitalValue = {"portset_name": "DvsPortset-0", "name": "DVS_NAME"}
  CachedValue =
 DesiredValue =
     Revision = 3

 

Resolution

Powered by Wolken Software