VM becomes unresponsive when tasks such as vMotion, clone or create VM run within a vSphere environment

Products

• VMware vSphere ESXi

Issue/Introduction

• If tasks or operations such as vMotion, clone or create VM are performed within the environment, a particular VM with no association to the task will report the guest OS as hung with vmware.log messages such as below:

YYYY-MM-DDTHH:MM:SS.Z| vcpu-0| | I005: Tools: [RunningStatus] Last heartbeat value 604585 (last received 21s ago)
YYYY-MM-DDTHH:MM:SS.Z| vmx| | I005: GuestRpcSendTimedOut: message to toolbox timed out.
YYYY-MM-DDTHH:MM:SS.Z| vmx| | I005: GuestRpc: app toolbox's second ping timeout; assuming app is down
YYYY-MM-DDTHH:MM:SS.Z| vmx| | I005: Tools: [AppStatus] Last heartbeat value 604585 (last received 21s ago)

• As the OS is hung, it will stop responding to the console or SSH/RDP sessions.
• Ping requests will drop as well.

Note: This could be a particularly odd behavior, hard to identify and diagnose.

Environment

• VMware vSphere ESXi 6.5
• VMware vSphere ESXi 6.7
• VMware vSphere ESXi 7.x

Cause

The virtual machine memory limit is set to "0" in the vSphere Client, meaning the virtual machine would not get any memory.

To start verifying this behavior, check the vmware.log of the virtual machines with the problems and look up for a similar message:

YYYY-MM-DDTHH:MM:SS.Z| vmx| opID=HB-SpecSync-host-56@3762046-#######-94-##b3| A000: ConfigDB: Unsetting "sched.mem.max"

Or check the virtual machine configuration in the vSphere Client, under the Memory configuration, confirm that the Limit is set to "0"

Resolution

To fix this issue, avoid setting a memory limit of a virtual machine to "0". To change the memory limit:

1. Open the vSphere Client.
2. Right click the virtual machine with the problem and then click "Edit Settings".
3. Expand the "Memory" section.
4. Change the Limit from "0" to "Unlimited".
   If you have a business case and need a memory limit, our recommendation is to set it to any number greater than the minimum memory required for the Guest Operating system + the Applications running inside the virtual machine to be able to run correctly.
5. Click "Ok" to save the settings.

Additional Information

Impact/Risks:
This could potentially cause management problems of the vSphere environment, since almost any action will impact the affected production virtual machine causing downtime, preventing organizations to manage vSphere during production hours.
For example, while doing vMotions, cloning virtual machines or creating new virtual machines.