Cisco HyperFlex VM Power On Failure Out of Resources and NFSLock I/O error

Products

• VMware vSphere ESXi

Issue/Introduction

• A VM located on Cisco Hyperflex fails to boot with error: "A general system error occurred: Launch failure ... Out of resources Cause: Out of resources"

• The ESXi host logs display I/O errors:
  
  [from /var/log/vmkernel.log]
  
  

  vmkernel: cpu52:2101472)NFSLock: 5141: failed to get lock on file protectedlist(.lck-################) 0x430b4a7d0ae0 on ######## ...  I/O error. fromunlink N

 

• Manual attempts to create empty files on the Cisco HyperFlex datastore fail with I/O errors.

Environment

• VMware ESXi 8.x
• VMware vCenter 8.x
• Cisco Hyperflex

Cause

• A critical resource exhaustion or health state on the underlying storage backend (e.g., Cisco HyperFlex) preventing the creation of .lck files.
  • NOTE: While the "failed to get lock on file protectedlist" can indicate other file locking issues, the key point is the fact it returned an I/O error, rather than "Busy". In this case, this is a 3rd party issue related to the Cisco Hyperflex storage backend. 

Resolution

• Contact Cisco to investigate the storage backend on the Hyperflex.

Additional Information

• For more information on troubleshooting locked files on NFS datastores, see KBs:
  • Cannot power on a virtual machine located in an NFS datastore
  • Investigating Virtual Machine file locks on ESXi Host(s)