VM becomes unresponsive when tasks as vMotion, clone or create VM run within a vSphere environment
search cancel

VM becomes unresponsive when tasks as vMotion, clone or create VM run within a vSphere environment

book

Article ID: 318704

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
If tasks or operations such as vMotion, clone or create VM are performed within the environment, a particular VM with no association to the task will report the guest OS as hung with vmware.log messages such as below:
 

2022-03-29T12:07:02.097Z| vcpu-0| | I005: Tools: [RunningStatus] Last heartbeat value 604585 (last received 21s ago)
2022-03-29T12:07:02.109Z| vmx| | I005: GuestRpcSendTimedOut: message to toolbox timed out.
2022-03-29T12:07:02.109Z| vmx| | I005: GuestRpc: app toolbox's second ping timeout; assuming app is down
2022-03-29T12:07:02.109Z| vmx| | I005: Tools: [AppStatus] Last heartbeat value 604585 (last received 21s ago)
  • As the OS is hung it will stop responding console or SSH/RDP sessions.
  • Ping requests will drop as well.

Note: This could be a particularly odd behavior, hard to identify and diagnose.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.x
VMware vSphere ESXi 6.5

Cause

The virtual machine memory limit is set to "0" in the vSphere Client, meaning the virtual machine would not get any memory.

To to start verifying this behavior, check the vmware.log of the virtual machines with problems and look up for a similar message:
 

2022-03-29T12:06:34.671Z| vmx| opID=HB-SpecSync-host-56@3762046-#######-94-##b3| A000: ConfigDB: Unsetting "sched.mem.max"


Or check the virtual machine configuration in the vSphere Client, under the Memory configuration, confirm that the Limit is set to "0"

Resolution

To fix this issue, avoid setting a memory limit of a virtual machine to "0". To change the memory limit:
  1. Open the vSphere Client.
  2. Right click the virtual machine with problems and then click "Edit Settings".
  3. Expand the "Memory" section.
  4. Change the Limit from "0" to "Unlimited".
    1. if you have business case and need a memory limit, our recommendation is to set it to any number greater than the minimum memory required for the Guest Operating system + the Applications running inside the virtual machine to be able to run correctly.
  5. Click "Ok" to save the settings.


Additional Information




Impact/Risks:
This could potentially cause management problems of the vSphere environment, since almost any action will impact the affected production virtual machine causing downtime, preventing organizations to manage vSphere during production hours, for example, while doing vMotions, cloning virtual machines or creating new virtual machines.