VM Performance Issues - vmware.log flooded with "Receiving GuestStats.SetNotificationTime request"

Products

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• VM is facing high CPU utilizations.
• Continuous Network latency issues.
• VM is not stable.

 

From vmware.log of the VM, you will see the following messages flooded.

 

2024-05-21T09:42:49.783Z In(05) vmx - VigorTransportProcessClientPayload: opID=ProcessScheduledTaskFiring-3113c398-01-b6-7397 seq=928757: Receiving GuestStats.SetNotificationTime request.
2024-05-21T09:42:49.783Z In(05) vmx - VigorTransport_ServerSendResponse opID=ProcessScheduledTaskFiring-3113c398-01-b6-7397 seq=928757: Completed GuestStats request.
2024-05-21T09:43:09.784Z In(05) vmx - VigorTransportProcessClientPayload: opID=ProcessScheduledTaskFiring-3113c398-01-b6-7397 seq=928781: Receiving GuestStats.SetNotificationTime request.

 

Note: Log excerpts are examples. Date, time, and environmental variables may vary depending on your environment.

Cause

• These lines are related to the "GuestIntrospection" feature in VMware.
• This feature allows the ESXi host to gather performance and health statistics from the guest VMs.
• Specifically, these entries show the VMs sending a GuestStats.SetNotificationTime request to the ESXi host.
• This request informs the host how often the VM wants to send its performance data.
• This behavior is observed on Host that has issue with syslog-forwarding.
• Issue occurred due to log level from syslog is not limited.

Resolution

Run the following commands in the ESXi host where the problematic VM is present.

These commands will limit the syslog size.

 

esxcli system syslog config logfilter set --log-filtering-enabled=true


esxcli system syslog config logfilter add -f "0 | Hostd | < logRegexp >"


esxcli system syslog reload

 

No outage for the ESXi Host.

Additional Information

We executed the commands & monitored the VM for 24 hours. Found that the logs are not repeated & VM started performing normal.