VCF Administrator is Unable to Migrate Offline Servers to New Cluster with Error "File Not Found"

Products

• VMware vSphere ESXi

Issue/Introduction

• VVOL objects appear to be unavailable following upgrade of HPE OS on the storage array.
• A Review of /var/run/log/vpxa.log on the esxi host, the following error is reported, which points to the mapping to object no longer being present.
  
  [VpxLRO] -- ERROR vpxa -- vpxapi.VpxaService.search
Datastore: :vim.fault.FileNotFound
Vpxa[2099709] --> Result:
Vpxa[2099709] --> (vim.fault.FileNotFound) {
 -->    faultCause = (vmodl.MethodFault) null,
 -->    faultMessage = <unset>,
 -->    file = "ds:///vmfs/volumes/vvol:####/naa.####"
-->    msg = ""
--> }
--> Args:
-->
--> Arg url:
--> "ds:///vmfs/volumes/vvol:####/naa.####"
--> Arg recursive:
  
  
• Additionally /var/run/log/vvold.log on the esxi host reports the following error:
  
  VVold[####] [Originator@6876 sub=Default IpcCtx=####] SI::QueryVirtualVolumeInt Empty response
VVold[####] [Originator@6876 sub=IpcSvc IpcCtx=####] HandleRequest: Error handling function QueryUuidForFriendlyName: Empty response

Environment

VMware ESXi 8.0.3 build-24585383

HPE VVOL

Cause

The above 'empty response' message indicates the expected data path is no longer present from Virtual Center's perspective.

This is due to a communication problem following the HPE SAN OS upgrade, as the VASA API is no longer on appropriate version matching the array side VASA. This version mismatch leads to the communication issue.

Resolution

To resolve the VASA API version mismatch in the Virtual Center, please follow the steps below:

1. In Virtual Center, navigate to Storage providers
   
   
   
2. Remove the storage provider
3. Add the storage provider back.
   
   At this point Virtual Center Server will detect the new VASA API version on the VASA provider server (Array Side). 

Manage Storage Providers for VVOL

Additional Information

Please Note: If the VASA certificate is expired, ensure the certificate is refreshed/renewed after re-adding the storage provider.