Virtual Machines on one ESXi host shutdown accidentally with MCE errors

Products

• VMware vSphere ESX 7.x

Issue/Introduction

• Virtual Machines shutdown in one ESXi host
• MCE errors observed just before VM shutdown in vmkernel.log


  Memory Controller Scrubbing Error on Channel 1

 
• Sorage failure logs comes right after MCE error in vmkernel.log

  
  Lost access to volume <volume UUID> due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.


• VM shutdown due to storage issues in vmware.log
  
 [msg.hbacommon.locklost] The lock protecting '<vm-name>.vmdk' has been lost, possibly due to underlying storage issues.
  PANIC: Exiting because of failed disk operation.

Environment

VMware vSphere ESXi 7.x

Cause

Memory controller errors can trigger a cascade of storage I/O issues, and shutdown VM afterwards.

Resolution

Please reach to hardware vendor to check memory issue further.

For further explanation of MCE errors, please refer to KB:
ESXi Host Becomes Unresponsive Due to Memory Controller Errors Leading to Storage I/O Issues