VPXD utilization spike to 100% while initiating the reconnect task

Products

• VMware vCenter Server 8.0

Issue/Introduction

While connecting ESXi to vCenter VPXD utilization spike to 100% CPU

Log location: /var/log/vmware/vpxd

[YYYY-MM-DDTHH:MM:SS] info vpxd[640610] [Originator@6876 sub=vpxLro opID=<OP-ID>] [VpxLRO] -- BEGIN task-50494050 -- host-501XX1 -- vim.HostSystem.reconnect -- 52XX13X0-2bXX-f307-c859-XXb579ec7X36(52dXXfcc-ca2f-5X90-XXf2-Xd552e4XX1e9)
[YYYY-MM-DDTHH:MM:SS] info vpxd[640610] [Originator@6876 sub=HostCnx opID=<OP-ID>] [VpxdHostCnx::AddConnection] cnx: XXXXXXXX-a372-XXXX-063f-33e030cXXXXX, h:
[YYYY-MM-DDTHH:MM:SS] info vpxd[640610] [Originator@6876 sub=HostDlfDownloader opID=<OP-ID>] Skip downloading DLFs for Host 'VMware ESX Server', Version: '8.0'File Version: '8.0.2.0', DLF Directory Location: '/etc/vmware-vpx//licenses/site//VMware ESX Server/8.0/8.0.2.0'
[YYYY-MM-DDTHH:MM:SS] info vpxd[640934] [Originator@6876 sub=MoLicenseDataMgr opID=<OP-ID>] [LicenseDataManager::Processor] Processed auto-assigned licenses for host "ESXi_FQDN"
[YYYY-MM-DDTHH:MM:SS] info vpxd[640610] [Originator@6876 sub=InvtHostCnx opID=<OP-ID>] Set server; host(esx): XX.XX.XX.XX, server(vc): XX.XX.XX.XX, serverId: XX53XX1, host key: XXXXXXXX-XXXX-XXXX-XXXX-33e030cXXXXX
[YYYY-MM-DDTHH:MM:SS] info vpxd[640610] [Originator@6876 sub=Vsan opID=<OP-ID>] Destroying VSAN dynamic MOs for host-Mo_ID.
[YYYY-MM-DDTHH:MM:SS] info vpxd[640610] [Originator@6876 sub=Vsan opID=<OP-ID>] Done creating VSAN dynamic MOs for Host-Mo_ID.

Cause

vpx_host table has a null value for the host in question under product_version.

Resolution

This is a known issue .Currently there is no resolution

To workaround follow below steps.

Take a Backup of the vcdb and a snapshot of the vCenter machine. 

 

ssh to vCenter appliance

 

1. Stop vpxd

    /usr/lib/vmware-vmon/vmon-cli --stop vpxd

 

2. Take backup of VCDB

    /opt/vmware/vpostgres/current/bin/pg_dump -U postgres VCDB > /tmp/VCDBBackup1

 

3. Connect to vCenter database

/opt/vmware/vpostgres/current/bin/psql -h 127.0.0.1 -U postgres VCDB

 

4. To identify the host that has product_version missing.

    select dns_name, id, product_version from vpx_host;

 

3. Update the table VPX_HOST with product_version.

    a. update vpx_host set product_version = "ESXi version'8.0.1.0'" where id = 501761;

    b. select product_version from vpx_host where id = 501761;

    c. quit

 

4. Start vpxd

    /usr/lib/vmware-vmon/vmon-cli --stop vpxd

 

5. Reconnect the host from VC UI

 

NOTE: product_version should be of current ESXI version.

 

Example: ESXi 8.0.1.0

Additional Information

In this case its ESXi version 8.0.1.0