(/vmfs/volumes/datastore/vm name/vmware.log) indicate VMware Tools heartbeat timeouts and internal RPC failures while the guest is hung:YYYY-MM-DDThh:mm:ss| vmx| | I005: GuestRpcSendTimedOut: message to toolbox timed out.YYYY-MM-DDThh:mm:ss| vmx| | I005: Tools: [AppStatus] Last heartbeat value #### (last received 20s ago)YYYY-MM-DDThh:mm:ss| vmx| | I005: GuestRpc: app toolbox's second ping timeout; assuming app is downYYYY-MM-DDThh:mm:ss| vcpu-0| | I005: Tools: Tools heartbeat timeout.YYYY-MM-DDThh:mm:ss| vmx| | I005: Guest: *** WARNING: GuestInfo collection interval longer than expected; actual=218 sec, expected=30 sec. ***
/var/log/messages shows RCU scheduling stalls followed by soft lockups on one or more CPUs, specifically involving sapstartsrv or kworker processes:MM-DDThh:mm:ss #### kernel: rcu: INFO: rcu_sched self-detected stall on CPUMM-DDThh:mm:ss #### kernel: rcu: Possible timer handling issue on cpu=1MM-DDThh:mm:ss #### kernel: rcu: rcu_sched kthread starved for #### jiffies! ->cpu=1MM-DDThh:mm:ss #### kernel: watchdog: BUG: soft lockup - CPU#1 stuck for ####s! [sapstartsrv:####]MM-DDThh:mm:ss #### kernel: watchdog: BUG: soft lockup - CPU#0 stuck for ####s! [kworker/0:1H:####]
VMware vSphere ESXi
A definitive root cause cannot be established at the hypervisor layer. The logs indicate an internal guest OS unresponsiveness, specifically an RCU (Read-Copy-Update) scheduling stall and soft lockup involving the sapstartsrv process. These events occurred without correlated errors or interruptions from the ESXi hypervisor.
Follow these steps to eliminate host-level issues:
Note: Perform these validation steps sequentially to rule out underlying infrastructure factors. Once it has been verified that host CPU and storage are not the cause, proceed to engage the OS vendor.
esxtop. Press 'u' to switch to the disk device view./var/run/log/vmkernel.log for warnings such as: performance has deteriorated. I/O latency increased.[YYYY-MM-DDTHH:MM:SS] cpu51:#)WARNING: ScsiDeviceIO: 513: Device naa.########## performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds.[YYYY-MM-DDTHH:MM:SS] cpu47:#)WARNING: ScsiDeviceIO: 1443: Device naa.######### performance has deteriorated. I/O latency increased from average value of 12017 microseconds to 254228 microseconds.Soft lockup and rcu_sched can srall for multiple reasons. One such example can be seen from vendor documentation: System hangs with soft lockup and rcu_sched CPU stall events
KB reference to troubleshoot high latency issues: Multiple VMs experience latency issues.