Unable to power-on a virutal machine on VMFS datastore with "NVRAM: write failed"

Products

VMware vSphere ESXi

Issue/Introduction

Unable to power-on a virutal machine on VMFS datastore with the following event:

Error message on <VM_NAME> on <HOST_NAME> in ha-datacenter: NVRAM: write failed.

Symptoms:

Errors observed on an ESXi host on that the affected virtual machine is deployed.

hostd.log:
YYYY-MM-DDThh:mm:ss.###Z info hostd[2103629] [Originator@6876 sub=Libs opID=########-#####-auto-####-h5:########-##-##-##-###] DictionaryLoad: Cannot open file "/vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx": Input/output error.
YYYY-MM-DDThh:mm:ss.###Z info hostd[2103629] [Originator@6876 sub=Libs opID=########-#####-auto-####-h5:########-##-##-##-###] VigorOffline_GenSecPolicy: retry reading /vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx
YYYY-MM-DDThh:mm:ss.###Z verbose hostd[2103629] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx opID=########-#####-auto-####-h5:########-##-##-##-###] Generate policy from cfg message: Unable to load configuration file '/vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx'.
YYYY-MM-DDThh:mm:ss.###Z error hostd[2103629] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx opID=########-#####-auto-####-h5:########-##-##-##-###] Could not perform config check (storage accessible): Fault cause: vim.fault.GenericVmConfigFault
YYYY-MM-DDThh:mm:ss.###Z info hostd[2103629] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx opID=########-#####-auto-####-h5:########-##-##-##-###] VM state has changed; config check to be retried.
YYYY-MM-DDThh:mm:ss.###Z verbose hostd[2102715] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/########-########-####-############/<VM_NAME>/<VM_NAME>.vmx opID=########-#####-auto-####-h5:########-##-##-##-####] Handling vmx message 7273: Input/output error
--> NVRAM: write failed.
YYYY-MM-DDThh:mm:ss.###Z info hostd[2099508] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=########-#####-auto-####-h5:########-##-##-##-###] Event ###### : Error message on <VM_NAME> on <HOST_NAME> in ha-datacenter: NVRAM: write failed.

Environment

VMware ESXi

Cause

Slow response followed by "No connection" from an underlying storage LUN of the affected VMFS datastore may cause the issue.

vmkernel.log:
YYYY-MM-DDThh:mm:ss.###Z cpu20:2097916)WARNING: ScsiDeviceIO: 1513: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 1687 microseconds to 33751 microseconds.
YYYY-MM-DDThh:mm:ss.###Z cpu32:2097918)WARNING: ScsiDeviceIO: 1513: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 1687 microseconds to 68851 microseconds.
YYYY-MM-DDThh:mm:ss.###Z cpu36:2097920)WARNING: ScsiDeviceIO: 1513: Device naa.################################ performance has deteriorated. I/O latency increased from average value of 1688 microseconds to 34124 microseconds.

YYYY-MM-DDThh:mm:ss.###Z cpu12:2097483)ScsiDeviceIO: 12480: Task mgmt request issued to device naa.################################ is stuck (WorldID ########, Cmd 0x93, CmdSN 3dba8dc). Issuing yellow notification to the application
YYYY-MM-DDThh:mm:ss.###Z cpu11:2097483)ScsiDeviceIO: 12527: Task mgmt request issued to device naa.################################ is stuck (WorldID ########, Cmd 0x93, CmdSN 3dba8dc). Issuing red notification to the application
YYYY-MM-DDThh:mm:ss.###Z cpu7:2103163)ALERT: BC: 3177: File protectedlist closed with dirty buffers. Possible data loss. YYYY-MM-DDThh:mm:ss.###Z cpu33:2097465)ScsiVmas: 1074: Inquiry for VPD page 00 to device naa.################################" failed with error No connectionYYYY-MM-DDThh:mm:ss.###Z cpu7:13944452)WARNING: ScsiDeviceIO: 12155: READ CAPACITY on device "naa.################################" from Plugin "NMP" failed. I/O error
YYYY-MM-DDThh:mm:ss.###Z cpu34:2097983)ScsiDeviceIO: 7122: Waited for completion for all issued commands for partition naa.################################:1, for 120 secs. 1 completions still awaited. Giving up !

Issued SCSI commands (not only READ, WRITE but also VAAI WRITE SAME / 0x93, INQUIRY, READ CAPACITY) got stuck and failed with error "No connection".
VMFS operations related to the virtual machine power-on failed consequently.
Looks saturated performance on the target LUN.

Resolution

Rebooting the affected ESXi host might be workaround.
Engage a storage support for RCA and prevention of recurrence.

Additional Information

Disabling Hardware Accelerated Init (WRITESAME) in ESXi