Linux VMs intermittently lose their IP addresses

Products

• VMware vSphere ESXi

Issue/Introduction

• The VM's performance chart in vSphere may show a CPU spike, no more disk activity, and a lower memory usage.
• Aria Operations may show a surge in capacity alerts around the time that the CPU appears to spike on the VM.
• The VM itself shows in a NotReady state and no longer shows an IP address in the web UI.
• The vmware.log of the VM may show the NIC being quiesced, and then indication that the VMware Tools stopped responding shortly after
  2025-11-16T20:03:30.611Z In(05) vcpu-1 - VMXNET3 user: Quiesce device 0.
2025-11-16T20:04:03.423Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
2025-11-16T20:04:03.424Z In(05) vmx - Tools: [AppStatus] Last heartbeat value 52893 (last received 6s ago)
2025-11-16T20:04:03.424Z In(05) vmx - TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
2025-11-16T20:04:18.072Z In(05) vcpu-0 - Tools: Tools heartbeat timeout.
2025-11-16T20:04:18.073Z In(05) vcpu-0 - Tools: Running status rpc handler: 1 => 0.
2025-11-16T20:04:18.073Z In(05) vcpu-0 - Tools: Changing running status: 1 => 0.
2025-11-16T20:04:18.073Z In(05) vcpu-0 - Tools: [RunningStatus] Last heartbeat value 52893 (last received 20s ago)
2025-11-16T20:04:23.425Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
2025-11-16T20:04:43.426Z In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.

Environment

ESXi 8.x
VMware vCenter 8.x

Cause

This is most likely caused by a backup solution quiescing the VMs during a backup window.  A guest OS and software application which is particularly busy may intermittently fail to make a timely response to a quiesce within the default time limit, preventing VMware Tools from successfully suspending I/O.

Resolution

1. Ensure VMware Tools is at the latest version that is supported by the ESXi Host version.
2. Engage with the backup team and application owner:
   • To determine if a quiesced backup is required.  Quiesced backups can be disabled for the right type of workloads.
   • If reducing the concurrent backups at any given time may improve the guest OS's response time to quiesce requests.