VM on vVol datastore become inaccessible after a power-off task

Products

• VMware vSphere ESXi

Issue/Introduction

• Previously to the issue, the VM was powered on and functioning well, but when a power-off task is triggered, the VM becomes inaccessible.
• A certificate update/change may have been performed on the VASA provider from the array side.
• The VASA provider is online on vCenter Server when checked through selecting the vCenter in the inventory > Configure > Storage Providers.
• There may be hosts that are not impacted in the cluster.
• In an SSH to the impacted host(s), the "Status: syncError" error is seen when checked with the command "esxcli storage vvol vasaprovider list" as in the example below:

[root@ESXI:~] esxcli storage vvol vasaprovider list

VP-NAME

   VP Name: VPNAME

   URL: https://###.###.##.## : #### 

   Status: syncError

   Arrays:

         Array Id: com.vp-storage:########-####-#####-####-############

         Is Active: true

         Priority: 200

 

• In the var/run/log/vvold.log, you may see messages as below:

[timestamp] info vvold[2099724] [Originator@6876 sub=Default opID=lro-#########################] VasaSession::Initialize url is empty
[timestamp] warning vvold[2099724] [Originator@6876 sub=Default opID=lro-#########################] VasaSession::DoSetContext: Empty VP URL for VP (###########-###)!
[timestamp] info vvold[2099724] [Originator@6876 sub=Default opID=lro-#########################] Initialize: Failed to establish connection https://VASA-IP:port-ID/version.xml
[timestamp] info vvold[2099724] [Originator@6876 sub=Default opID=lro-#########################] ScheduleEventPollLocked: ###########-### Unable to cancel timer, bailing out.
[timestamp] error vvold[2099724] [Originator@6876 sub=Default opID=lro-#########################] Initialize: Unable to init session to VP ###########-### state: 0
[timestamp] error vvold[4642825] [Originator@6876 sub=IO.Http opID=lro-#########################] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection timed out)

• When testing the network connectivity with the VASA provider IP and port ID from the impacted ESXi host(s), there is an error, as in the examples below:

1:

[root@ESXI:~] wget https://VASA-IP:port-ID/version.xml

Connecting to VASA-IP:port-ID (VASA-IP:port-ID)

wget: error getting response

 

2:

[root@ESXI:~] nc -z -v VASA-IP port-ID

nc: connect to VASA-IP port port-ID (tcp) failed: Connection timed out

 

 

Environment

VMware vSphere ESXi (All versions).

Cause

The pattern in the logs and the results of the wget and nc commands suggest issues with the communication of the impacted host(s) and the needed port ID in the VASA provider.

Resolution

• Check the firewall rules between the ESXi host(s) and the necessary VASA port ID and mitigate any communication issues between them.