vCenter Server shows Intermittent gaps in performance data for Esxi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Advanced performance charts shows gaps in performance charts, i.e Select host---> Advanced-->Last day     
• Some hosts in vCenter might have performance data
• Reviewing hostd, vmkernel logs will have below errors

/var/log/hostd.log

<YYYY-MM-DD>T<time> In(166) Hostd[2099896] -->          value = "ntpd"
<YYYY-MM-DD>T<time> In(166) Hostd[2099923] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 428 : ntpd stepped system clock to 1723895701.996297000, synchronization lost
<YYYY-MM-DD>T<time> In(166) Hostd[2099896] -->          value = "ntpd"
<YYYY-MM-DD>T<time> In(166) Hostd[2099922] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 476 : ntpd stepped system clock to 1723915261.211801000, synchronization lost
<YYYY-MM-DD>T<time> In(166) Hostd[2099896] -->          value = "ntpd"
<YYYY-MM-DD>T<time> In(166) Hostd[2099934] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 563 : ntpd stepped system clock to 1723953515.633264000, synchronization lost

/var/log/vmkernel.log

<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu41:2099397)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
2024-08-17T08:53:10.995Z Wa(180) vmkwarning: cpu27:2099397)WARNING: NTPClock: 1444: ntpd changed system clock to 1723884790.723765000, lost synchronization to upstream time servers
<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu27:2099397)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
<YYYY-MM-DD>T<time> In(182) vmkernel: cpu12:2986991)Deactivating Daemon ntpd.
<YYYY-MM-DD>T<time> In(182) vmkernel: cpu12:2986991)Daemon ntpd deactivated.
<YYYY-MM-DD>T<time> In(182) vmkernel: TSC: 440052 cpu0:1)BootConfig: 752: forceOSCGrantPCIEHotplug = FALSE (0)
<YYYY-MM-DD>T<time> In(182) vmkernel: 0:00:00:06.700 cpu0:2097152)VMKAcpi: 239: \_SB_.PC00.NTPK: found VMKAPI-capable ACPI PNP Device.
<YYYY-MM-DD>T<time> In(182) vmkernel: 0:00:00:06.706 cpu0:2097152)VMK init (130/190): NTPClock_Init
<YYYY-MM-DD>T<time> In(182) vmkernel: cpu0:2097152)SysInitTable: 113: Finished sysInit step: NTPClock_Init in 1625 us.
<YYYY-MM-DD>T<time> In(182) vmkernel: cpu36:2097792)SchedVsi: 2075: Group: host/vim/vmvisor/init/ntpd(804): min=0 max=-1 minLimit=-1, units: mb
<YYYY-MM-DD>T<time> In(182) vmkernel: cpu36:2097792)SchedVsi: 2075: Group: host/vim/vmvisor/ntpd(861): min=15 max=15 minLimit=-1, units: mb
<YYYY-MM-DD>T<time> In(182) vmkernel: [INFO] Getting realpath failed: /etc/ntp.conf

<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu2:2099399)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu3:2099399)WARNING: NTPClock: 1444: ntpd changed system clock to 1723895701.996297000, lost synchronization to upstream time servers
<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu3:2099399)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers
<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu4:2099399)WARNING: NTPClock: 1444: ntpd changed system clock to 1723915261.211801000, lost synchronization to upstream time servers
<YYYY-MM-DD>T<time> Wa(180) vmkwarning: cpu4:2099399)WARNING: NTPClock: 1691: system clock synchronized to upstream time servers

 

Cause

This issue occurs if vCenter is unable to poll the stats data from host due to various issues like NTP issues, host disconnect issues, host not responding.

Resolution

To fix this we can try below steps. 

• Review the statistics level is set to default  level of 1 on vCenter: 
• Check if vCenter, Hosts are in sync with NTP without any delays.