Determining the mapping between hostname and hostId in a VMware HA cluster

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article outlines steps to determine the hostname of an ESXi host from the hostId located in the log files.

Symptoms:

An unexpected failover in the vSphere HA (High Availability) cluster occurs.
The vpxd logs on vCenter Server contains entries referencing hostIds similar to:

Note: hostIds are host-1208 and host-1214.

On the vCenter server (vpxd.log):, located by default under /var/log/vmware/vpxd/

T16:48:45.796+01:00 [01724 verbose 'DAS' opID=SWI-d75d1509] [FdmManager::GetHostList] host list version for vim.ClusterComputeResource:domain-c9 (Cluster): 21
T16:48:45.796+01:00 [01724 verbose 'DAS' opID=SWI-d75d1509] [FdmManager::GetHostList] New host list is: (csi.HostList) {
dynamicType = <unset>,
faultDomainId = "44BA0E8A-D6E9-4902-86FE-A0BDE160ACF8-9-d409ae6-vCenter01",
version = 21,

info 'Default'] [VpxdMoHost::UpdateDasState] VC state for host host-1208 (initialized -> initialized), FDM state (Live -> FDMUnreachable), src of state (host-406 -> host-406)
info 'Default'] [VpxdMoHost::UpdateDasState] VC state for host host-1214 (initialized -> initialized), FDM state (Live -> FDMUnreachable), src of state (host-406 -> host-406)

The /var/log/fdm.log on ESXi hosts contains log entries referencing hostIds similar to:

T14:48:45.563Z [48D03B90 info 'Cluster' opID=SWI-d75d1509] [ClusterManagerImpl::StoreDone] Wrote hostlist version 21 verbose 'Cluster' opID=SWI-2bc801f9] [ClusterSlave::LiveCheck] Heartbeat still pending for slave @ host-1208
verbose 'Cluster' opID=SWI-2bc801f9] [ClusterSlave::LiveCheck] Heartbeat still pending for slave @ host-1214

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 　

Environment

VMware vCenter Server 7.0.x
VMware vSphere ESXi 7.0.x

VMware vCenter Server 8.0.x
VMware vSphere ESXi 8.0.x

Cause

This issue occurs when the vCenter Server and ESXi reference hostIds rather than hostnames.

Resolution

To resolve this issue, determine the hostId from the vpxd.log file or the fdm.log file, match the hostId in the logs with the ESXi hostnames.

To determine the hostname of an ESXi host from the hostId located in the vpxd.log files:

In the /var/log/vmware/vpxd/ log search for the Synchronizing host and the hostId.

For example:

T17:44:31.563Z [7FB34D20A700 info 'vpxdvpxdInvtHostCnx'] [VpxdHostSync] Synchronizing host: host-1208 of the host (hostname.domain.local, XX.XXX.XXX.XX)

Where host-1208 corresponds to server 12345.

When vCenter Server logs are not accessible, determine the hostname of an ESXi host from the hostId located in the fdm.log files. Run the prettyPrint.sh script with the hostlist parameters on the ESXi host to display FDM configuration.

To determine the hostname of an ESXi server from the hostId located in the fdm.log files:

Note: Troubleshooting and further analysis of logs is performed when VMware have the hostname. It is essential to gather all the ESXi host log files as soon as possible after a failure.

To determine the hostname of an ESXi server from the hostId located in the /var/log/fdm.log:

As user root on any host on the cluster open a Secure Shell (SSH) connection to the host.
Run the command:

/opt/vmware/fdm/fdm/prettyPrint.sh hostlist

Note: /etc/opt/vmware/fdm/hostlist is an xml file and contains information on the hostnames, IP Addresses, MAC addresses and heartbeat datastore of hosts participating in the cluster.
This is an example of the hostlist file, where you can determine the relationship between hostId and hostName, host-1208 from the vpxd / FDM logs corresponds to ESXi hostName hostname.domain.local.

Warning: The prettyPrint.sh script is only available to display information. Do not modify these files for any reason. The cluster configuration is performed exclusively by the vSphere Client or by PowerCLI.

<faultDomainId>44BA0E8A-D6E9-4902-86FE-A0BDE160ACF8-9-d409ae6-vcbr04</faultDomainId>
<version>21</version>
<hostId>host-1208</hostId>
<hostName>hostname.domain.local</hostName>
<sslThumbprint>21:E5:C9:24:9F:BC:CC:AF:84:E7:6A:95:02:4C:ED:22:8A:BD:D4:93</sslThumbprint>
<ipAddress>xx.xx.xx.xxx</ipAddress>
<mac>xx:xx:xx:xx:xx:xa</mac>
<mac>xx:xx:xx:xx:xx:xb</mac>
<mac>xx:xx:xx:xx:xx:xc</mac>
<mac>xx:xx:xx:xx:xx:xd</mac>
<mac>xx:xx:xx:xx:xx:xe</mac>
<mac>xx:xx:xx:xx:xx:xf</mac>
<mac>xx:xx:xx:xx:xx:xh</mac>
<mac>xx:xx:xx:xx:xx:xi</mac>
<heartbeatDatastore>/vmfs/volumes/5028de7e-36c10eaa-7037-0017a4770000</heartbeatDatastore>
<heartbeatDatastore>/vmfs/volumes/4fbf41e5-da3ba7c6-993c-0017a4770460</heartbeatDatastore>
<hostdPort>443</hostdPort>
<version>5.0.0</version>
<build>474610</build>

Note: The clusterconfig, compatlist and hostlist output of prettyPrint.sh have their own version mentioned between xml tags <version> </version>.
It is important to match the file version with the messages in the logs allowing for the time stamp in the fdm or vpxd logs.

Additional Information

On the ESXi host the /opt/vmware/fdm/fdm/prettyPrint.sh script displays information on the HA cluster. For example, the command syntax is:

prettyPrint.sh <hostlist|clusterconfig|compatlist|vmmetadata>

Note: Related FDM files are located in the directory /etc/opt/vmware/fdm.

Warning: The prettyPrint.sh script is only available to display information. Do not modify these files for any reason. The cluster configuration is performed exclusively by the vSphere Client or by PowerCLI.

For more information on collecting logs, see:

Additional references:

Impact/Risks:
Warning: The prettyPrint.sh script is only available to display information. Do not modify these files for any reason. The cluster configuration is performed exclusively by the vSphere Client or by PowerCLI.