VM shuts down or reboots unexpectedly during snapshot operation

Products

• VMware vSphere ESXi

Issue/Introduction

• VM shuts down unexpectedly during image-based backup snapshot request or during regular snapshot operation
• Previous and subsequent snapshots do not reproduce the issue 
• vmware.log indicates that the .nvram file is missing: 
  In(05) vcpu-0 68b329ab-38-2a03 SNAPSHOT: SnapshotGetFileContext: Failed to open object '/vmfs/volumes/<volume_id>/<vm_name>/<vm_name>.nvram': Could not find the file
..
In(05) vcpu-0 #### OBJLIB-FILEBE : FileBEOpen: can't open '<vm_name>.nvram' : Could not find the file (393218).
In(05) vcpu-0 #### NVRAMMGR:  NvmanReopen: Failed to reopen NVRAM: Could not find the file
In(05) vcpu-0 #### CPT: error syncing group NVMAN
..
In(05) vcpu-0 #### [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.
• After bringing the VM backup, the .nvram file is present in the VM directory 

Environment

VMware vSphere 8.x 

Cause

• The .nvram file of the VM was removed while the VMX process of the VM was still running 
• This may have been caused by someone manually removing the file or an automation script that removed the .nvram file while the VMX process was still running
• The snapshot process could not find the .nvram file and then the VM shuts down due to this unexpected condition

Resolution

• Review with your internal team if any automation or scripting is being run in the environment that may have removed the .nvram file from the VM directory 
• Add a sleep timer or a condition to check if the VMX process is still running in the script before proceeding to delete the .nvram file