vCenter Server shows intermittent gaps in performance data for ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Advanced performance charts shows gaps in performance charts     
• Some hosts in vCenter might have incomplete performance data
• The hostd and/or vmkernel logs may have the below errors

/var/log/hostd.log

[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099896] -->          value = "ntpd"
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099923] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 428 : ntpd stepped system clock to 1723895701.996297000, synchronization lost
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099896] -->          value = "ntpd"
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099922] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 476 : ntpd stepped system clock to 1723915261.211801000, synchronization lost
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099896] -->          value = "ntpd"
[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[2099934] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 563 : ntpd stepped system clock to 1723953515.633264000, synchronization lost

/var/log/vmkernel.log

[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu41:2099397)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
2024-08-17T08:53:10.995Z Wa(180) vmkwarning: cpu27:2099397)WARNING: NTPClock: 1444: ntpd changed system clock to 1723884790.723765000, lost synchronization to upstream time servers
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu27:2099397)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2986991)Deactivating Daemon ntpd.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2986991)Daemon ntpd deactivated.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: TSC: 440052 cpu0:1)BootConfig: 752: forceOSCGrantPCIEHotplug = FALSE (0)
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: 0:00:00:06.700 cpu0:2097152)VMKAcpi: 239: \_SB_.PC00.NTPK: found VMKAPI-capable ACPI PNP Device.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: 0:00:00:06.706 cpu0:2097152)VMK init (130/190): NTPClock_Init
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu0:2097152)SysInitTable: 113: Finished sysInit step: NTPClock_Init in 1625 us.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2097792)SchedVsi: 2075: Group: host/vim/vmvisor/init/ntpd(804): min=0 max=-1 minLimit=-1, units: mb
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2097792)SchedVsi: 2075: Group: host/vim/vmvisor/ntpd(861): min=15 max=15 minLimit=-1, units: mb
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: [INFO] Getting realpath failed: /etc/ntp.conf
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu2:2099399)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu3:2099399)WARNING: NTPClock: 1444: ntpd changed system clock to 1723895701.996297000, lost synchronization to upstream time servers

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.x

Cause

This issue occurs if vCenter is unable to poll stats data from the host due to various reasons such as NTP issues, host disconnect issues, or the host not responding.

Resolution

• Review the statistics level is set to default  level of 1 on vCenter
• Check if vCenter, Hosts are in sync with NTP