Power on VM is failing with failed swap file location can't access it

Products

• VMware vCenter Server

Issue/Introduction

Symptoms: 

• Power on VM is failing with failed swap file with unable to access the file
  
  
• Able to brows the datastore and see files under few VVOls but not all 
  
  
• ProtocolEndpoint is accessible from host: 
  
  [root@esxi-host:/vmfs/volumes/vvol:f9#############6-a6#############3] esxcli storage vvol protocolendpoint list
naa.2ff70002ac027e5b
Host Id: naa.2ff####2a#######
Array Id: com.hpe.primera:0x27###:array:0x27###
Type: SCSI
Accessible: true
Configured: true
Lun Id: naa.2ff####2a#######
  
  
• Storagecontainer is accessible 
  
  [root@esxi-host:/vmfs/volumes/vvol:f9#############6-a6#############3] esxcli storage vvol storagecontainer list
CBL-Primera-Vvol
StorageContainer Name: Vvol-01
UUID: vvol:f9#############6-a6#############3
Array: com.hpe.primera:0x27###:array:0x27###
Size(MB): 1######72
Free(MB): 3######2
Accessible: true
Default Policy:
  
  
• VASAprovider is online and accessible 
  
  [root@esxi-host:/vmfs/volumes/vvol:f9#############6-a6#############3] esxcli storage vvol vasaprovider list
Primera-Main
VP Name: Primera-Main
URL: https://##.###.###.##:9997/vasa
Status: online
Arrays:
Array Id: com.hpe.primera:0x27###:array:0x27###
Is Active: true
Priority: 255

• /var/run/logs/vvold.log shows createVirtualVolume call is successful 
  
  YYYY-MM-DDTHH:MM:SS info vvold[2099409] [Originator@6876 sub=Default] VASA task createVirtualVolume (id: 0x27E5B:ph:87455) SUCCEEDED in 203 ms (0 retries), new starting poll = 1000
  
• UpdateVirtualVolumeMetadata task successfully completed 
  
  YYYY-MM-DDTHH:MM:SS info vvold[2099819] [Originator@6876 sub=Default] SI UpdateVirtualVolumeMetadata successfully completed
  
• From vCenter UI vVOL Datastore was browsable and many of the vm folders shows required files
  
  
• The issue isolated for few volumes in the environment

Environment

vSphere ESXi 7.x
vSphere ESXi 8.x

 

Cause

Bind operation is failing on the storage, hence the created volume is not getting attached to the VM. Due to which the vm is failing to power on. 

• BindVirtualVolume failed as per /var/run/log/vvold.log

YYYY-MM-DDTHH:MM:SS info vvold[10154415] [Originator@6876 sub=Default] Came to SI::BindVirtualVolume: esxContainerId f9#############6-a6#############3  VVol Id naa.#000###0000000000#0#####000##### bindType Normal (isConfigVvol: true)
YYYY-MM-DDTHH:MM:SS info vvold[10154415] [Originator@6876 sub=Default] VasaOp::BindVirtualVolume [#1572014]: ===> Issuing 'bindVirtualVolume' to VP [Primera-Main:Connected (Outstanding 0/4)]
YYYY-MM-DDTHH:MM:SS error vvold[10154415] [Originator@6876 sub=Default] VasaOp::IsSuccessful [#1572014]: bindVirtualVolume transient failure: 22 (STORAGE_FAULT / Error has occurred. Details: Storage fault : Error: Host not found
-->  / )
YYYY-MM-DDTHH:MM:SS warning vvold[10154415] [Originator@6876 sub=Default] VasaOp[#1572014] ===> Transient failure bindVirtualVolume VP (Primera-Main) retry=false, batchOp=false container=f9#############6-a6#############3 timeElapsed=61 msecs (#outstanding 0)
YYYY-MM-DDTHH:MM:SS error vvold[10154415] [Originator@6876 sub=Default] VasaOp[#1572014] ===> FINAL FAILURE bindVirtualVolume, error (STORAGE_FAULT / Error has occurred. Details: Storage fault : Error: Host  not found
-->  / ) VP (Primera-Main) Container (f9#############6-a6#############3) timeElapsed=61 msecs (#outstanding 0)

Resolution

This issue requires additional analysis from the storage vendor to identify the cause and provide a permanent resolution.