Virtual machine experience network lost after MCE(Memory Controller Errors)

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

 

• The virtual machine experienced a network loss and may have subsequently rebooted, likely due to a Guest OS application triggers power off or reset.
• The virtual machine may also appear to have lost access or have slow response to the backing storage 
• ESXi vobd.log (/var/run/log/vobd.log)  shows many MCE and then datastore timeout

  ###Memory MCE

  YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [cpuCorrelator] 771462158768us: [vob.cpu.mce.log4] MCE bank 8: status:0x### misc:0x### addr:0x###### cpu:4 physAddr:0x###### physSize:0x40 ceCount:0x1
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [cpuCorrelator] 771846076344us: [vob.cpu.mce.log4] MCE bank 8: status:0x### misc:0x### addr:0x###### cpu:4 physAddr:0x###### physSize:0x40 ceCount:0x1
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [cpuCorrelator] 771854668561us: [vob.cpu.mce.log4] MCE bank 8: status:0x### misc:0x### addr:0x###### cpu:4 physAddr:0x###### physSize:0x40 ceCount:0x1
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [cpuCorrelator] 772512427258us: [vob.cpu.mem.ce.log] Corrected memory error at physAddr:0x######
 
 
###Datastore timeout
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [vmfsCorrelator] 772528184023us: [vob.vmfs.heartbeat.timedout] xxx-xxx-xxx-xxx <Datastore name>
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [vmfsCorrelator] 772524898774us: [esx.problem.vmfs.heartbeat.timedout] xxx-xxx-xxx-xxx <Datastore name>
 
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [vmfsCorrelator] 772532962565us: [vob.vmfs.heartbeat.timedout] yyy-yyy-yyy-yyy <Datastore name>
YYYY-MM-DDTHH:MM:SS In(14) vobd[2097668]:  [vmfsCorrelator] 772529397699us: [esx.problem.vmfs.heartbeat.timedout] yyy-yyy-yyy-yyy <Datastore name>

• Virtual machine non-responsive and then rebooted is observed from vmware.log
  YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - Tools: Tools heartbeat timeout.
YYYY-MM-DDTHH:MM:SS In(05)+ vcpu-0 - The CPU has been disabled by the guest operating system. Power off or reset the virtual machine.

Environment

ESXi 7.x/8.x

Cause

Hardware issue which leads to virtual machine work improperly.

 

 

Resolution

Please contact physical server vendor to further check physical memory