vCenter Server UI Slowness, Task Timeouts, and vpxd Service Crashes
search cancel

vCenter Server UI Slowness, Task Timeouts, and vpxd Service Crashes

book

Article ID: 404838

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

  • vCenter Server experiences significant slowness while loading inventory objects. Tasks take an extended time to complete or time out,

  • In vCenter, /var/log/vmware/vpxd/vpxd.log, reveals entries similar to the following, where exceptionally high [DoHostSyncTime] values prior to a vpxd service crash indicate that slow database query executions and host synchronization timeouts are directly causing the observed UI slowness and task failures by forcing the service to wait for database writes to complete before updating the inventory view.

[YYYY-MM-DDTHH:MM:SS] error vpxd[07403] [Originator@6876 sub=Default opID=HB-host-<id>@192###-####7b0] [VdbStatement::PrepareHandle] Error on preparing statement UPDATE VPX_HOST SET ROOT_RES_CONFIG=? WHERE ID=?
[YYYY-MM-DDTHH:MM:SS] warning vpxd[64861] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@104###-#####4cd] DoHostSync:host-<id> [ProcessChanges] took 14918751 ms
[YYYY-MM-DDTHH:MM:SS] warning vpxd[64861] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@104###-#####4cd] DoHostSync:host-<id> [DoHostSyncTime] took 14918923 ms
[YYYY-MM-DDTHH:MM:SS] warning vpxd[64861] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@104###-#####4cd] InvtHostSyncLRO::StartWork [HostSyncTime] took 14918925 ms
[YYYY-MM-DDTHH:MM:SS] warning vpxd[64861] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@104###-#####4cd] VpxLro::LroMain [TotalTime] took 1491892 ms


[YYYY-MM-DDTHH:MM:SS] error vpxd[64501] [Originator@6876 sub=Default opID=HB-host-<id>@30103-7a74fe6c] [VdbStatement::PrepareHandle] Error on preparing statement 
[YYYY-MM-DDTHH:MM:SS] warning vpxd[07403] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@192###-###7b0] DoHostSync:host-<id> [ProcessChanges] took 15947211 ms
[YYYY-MM-DDTHH:MM:SS] warning vpxd[07403] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@192###-###7b0] DoHostSync:host-<id> [DoHostSyncTime] took 15947242 ms
[YYYY-MM-DDTHH:MM:SS] warning vpxd[07403] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@192###-###7b0] InvtHostSyncLRO::StartWork [HostSyncTime] took 15947243 ms
[YYYY-MM-DDTHH:MM:SS] warning vpxd[07403] [Originator@6876 sub=VpxProfiler opID=HB-host-<id>@192###-###7b0] VpxLro::LroMain [TotalTime] took 1594724 ms
[YYYY-MM-DDTHH:MM:SS] panic vpxd[07403] [Originator@6876 sub=vpxCommon opID=HB-host-<id>@192###-###7b0] Unrecoverable VmRootError: 00007f2fb4085670 Backtrace: 

<Backtrace>

[YYYY-MM-DDTHH:MM:SS] error vpxd[07403] [Originator@6876 sub=Default opID=HB-host-<id>@192###-####7b0] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) ODBC error: () - 
[YYYY-MM-DDTHH:MM:SS] warning vpxd[06593] [Originator@6876 sub=VpxProfiler opID=CdrsLoadBalancer-57b90e8b] Time taken for DRS region [RetrieveHostVmInfo] took 1408195 ms
[YYYY-MM-DDTHH:MM:SS] info vpxd[06906] [Originator@6876 sub=drmLogger opID=CdrsLoadBalancer-73197e7b] No stats from host [vim.HostSystem:host-<id>,<host-fqdn>] for 10 minutes. Ignoring
[YYYY-MM-DDTHH:MM:SS]info vpxd[06618] [Originator@6876 sub=drmLogger opID=CdrsLoadBalancer-7b039e9c] No stats from host [vim.HostSystem:host-<id>,<host-fqdn>] for 10 minutes. Ignoring
[YYYY-MM-DDTHH:MM:SS] warning vpxd[64486] [Originator@6876 sub=MoHost opID=221a2071] The RPC for host host-<id> did not return in time

 

  • On the ESXi host running the vCenter Server VM, /var/run/log/vmkernel.log displays comprehensive Native Multipathing (NMP) failures, aborted commands, and SCSI device resets—indicating that the underlying storage array is failing to acknowledge write requests in a timely manner, as demonstrated by the following entries:

[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x8a (0x45d997204cc8, 4667171) to dev "naa. 600#####################0d5" on path "vmhba#:CO:T1:#" Failed:
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)NMP: nmp_ThrottleLogForDevice: 3875: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId. initiator-0x430d6825dfc0 CmdsN 0xffffc78f4f06cc70
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)WARNING: NMP: nmp_DeviceRequestFastDeviceprobe:237: NMP device "naa. 600#####################0d5" state in doubt; requested fast path state update ...
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139) ScsiDeviceIO: 4115: Cmd(0x45d997204cc8) 0x8a, CmdSN 0xffffc78f4f06cc70 from world 4667171 to dev "naa. 600#####################0d5" failed H:0x5 D:0x0 P:0x0
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)PVSCSI: 1688: ctx=0xffffc78f4f06cc70 cdb0=0x2a on scsi0:0 completed during reset (0x8)
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139) ScsiDeviceIO: 4115: Cmd(0x45d9a1579d48) Ox8a, CmdSN Oxffffc78f4f06a5f0 from world 4667171 to dev "naa. 600#####################0d5" failed H:0x5 D:0x0 P:0x0
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139) PVSCSI: 1688: ctx-0xffffc78f4f06a5f0 cdb0-0x2a on scsi0:0 completed during reset (0x8)
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl_handler : 1809: io_req: 0x45d98fc12f00 sc: 0x45d9972470c8 tag: 0x733 CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS Pending hdr status: FCPIO_ABORTED scsi_status: 0xO$
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_SUCCESS
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2405: io_req: 0x45d98fc12f00 sc: 0x45d9972470c8 id: 0x733 CMD_FLAGS: 0x73 CMD_STATE: FNIC_IOREQ_ABTS_PENDINGhdr status: FCPIO_SUCCESS ABTS cmpl received
[YYYY-MM-DDTHH:MM:SS] cpu35:2098139)ScsiDeviceIO: 4115: Cmd(0x45d9972470c8) 0x8a, CmdSN Oxffffc78f4f011dd0 from world 4667171 to dev "naa. 600#####################0d5" failed H:0x5 D:0x0 P:0x0

 

 

Cause

The ESXi host hosting the vCenter Server VM is experiencing high disk latency due to underlying storage connectivity issues. This latency severely impacts vCenter PostgreSQL database (VCDB) read/write operations, leading to delayed database transactions, vpxd service instability, missed ESXi host heartbeats, and degraded vCenter UI performance.

Resolution

  • Establish an SSH session to the ESXi host where the vCenter Server VM resides.
  • Run the esxtop command to monitor storage performance in real-time.
  • Press u to switch to the disk device view (Storage list).
  • Review the Avg. Write Latency columns. Specifically, observe the DAVG/cmd (Device Average Latency) metrics. Values consistently exceeding 25 milliseconds (ms) for sustained periods confirm that the underlying storage array is responding slowly to I/O requests.
  • Engage the storage vendor to validate the Storage Area Network (SAN) fabric health, path stability, and error counters to remediate the latency.
Powered by Wolken Software