vCenter Server shows gaps in performance data for ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Advanced performance charts shows gaps in performance data     
• Some hosts in vCenter may have incomplete performance data
• The hostd logs and/or vmkernel logs may have the following errors

/var/log/hostd.log
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099896] -->          value = "ntpd"
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099923] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 428 : ntpd stepped system clock to 1723895701.996297000, synchronization lost

/var/log/vmkernel.log
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu41:2099397)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
2024-08-17T08:53:10.995Z Wa(180) vmkwarning: cpu27:2099397)WARNING: NTPClock: 1444: ntpd changed system clock to 1723884790.723765000, lost synchronization to upstream time servers
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu27:2099397)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2986991)Deactivating Daemon ntpd.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2986991)Daemon ntpd deactivated.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: TSC: 440052 cpu0:1)BootConfig: 752: forceOSCGrantPCIEHotplug = FALSE (0)
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: 0:00:00:06.700 cpu0:2097152)VMKAcpi: 239: \_SB_.PC00.NTPK: found VMKAPI-capable ACPI PNP Device.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: 0:00:00:06.706 cpu0:2097152)VMK init (130/190): NTPClock_Init
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu0:2097152)SysInitTable: 113: Finished sysInit step: NTPClock_Init in 1625 us.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2097792)SchedVsi: 2075: Group: host/vim/vmvisor/init/ntpd(804): min=0 max=-1 minLimit=-1, units: mb
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2097792)SchedVsi: 2075: Group: host/vim/vmvisor/ntpd(861): min=15 max=15 minLimit=-1, units: mb
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: [INFO] Getting realpath failed: /etc/ntp.conf
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu2:2099399)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu3:2099399)WARNING: NTPClock: 1444: ntpd changed system clock to 1723895701.996297000, lost synchronization to upstream time servers

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.x

Cause

The issue occurs if vCenter is unable to poll stats data from the host due to various reasons; such as NTP issues, host disconnect issues, or the host in a Not Responding state.

Resolution

• Confirm the Statistics Level is set to the default of Level of 1 on the vCenter
• Check if the vCenter and Hosts are in sync with NTP