Various VM operations (VM Migration, snapshot operations, VM power on, etc) fail on VVOL datastore

Products

• VMware vCenter Server

Issue/Introduction

• DRS and manual vMotion is failing with error "The object or item referred to could not be found "

• VMs power on operations fails with error "Module VMMon power on failed."

• VM Snapshot operations also fail

• Couldn't browse the VVOL datastore.

• The existing virtual machines continue to operate normally.

Environment

vSphere 8.0.3

Cause

It was identified that out of the three Protocol Endpoints, one shows its accessible status as false. The Protocol Endpoint in question is no longer in use and had previously been deleted by the user.

Resolution

1. Unmap all the protocol endpoints including unused PE from storage array configuration and represent the working PE.

2. Perform Storage adaptor rescan at ESXi host level.  

3. VVOL datastore browse, VM Power On and migrate tasks should complete successfully.

Additional Information

Below are few commands to check the VVOL configuration while troubleshooting.

# esxcli storage vvol storagearray protocolendpoint list   - List containers or protocol endpoints on the storage array.

Example: 

xVP SCSI Array 1
   Name: xVP SCSI Array 1
   Array ID: com.vmware.vim:02004#######-1000000
   ProtocolEndpoints: 
      7a3941dc-eee7-69ca-3aba-############
         Vasa Id: 7a3941dc-eee7-69ca-3aba-############
         Host Id: eui.64323339########
         Type: SCSI
         Accessible: false
         Configured: false
         Lun Id: eui.64323339########
         Remote Host: 
         Remote Share: 
         NFS4x Transport IPs: 
         Server Scope: 
         Server Major: 
         Auth: 
         User: 

# esxcli storage vvol protocolendpoint list   -List all protocol endpoints that your host can access.

Example:

naa.624a9370b0005fb#################
   Host Id:
   Array Id: com.purestorage:b0005fb2-8f54-4181-b900-############
   Type: SCSI
   Accessible: false 
   Configured: false
   Lun Id: naa.624a9370b0005fb#################
   Remote Host:
   Remote Share:
   NFS4x Transport IPs:
   Server Scope:
   Server Major:
   Auth:
   User:
   Storage Containers: 5811f38b-faaf-335e-8f45-############

# esxcli storage vvol vasaprovider list -List all storage (VASA) providers associated with the host.

xVP
   VP Name: xVP
   URL:https://<VASA Provider IP address>:8443/vasa/version.xml 
   Status: syncError
   Arrays: 
         Array Id: com.vmware.vim:xxxxxxxx3e06-1000000
         Is Active: true
         Priority: 0

To rescan the Storage adaptors please refer to this KB- Performing a rescan of the storage on an ESXi host