vVOL Datastore becomes inaccessible following temporary loss of access to VP citing a STORAGE_FAULT

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

vvol datastore is inaccessible

vvold.log reports Storage Fault:

VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaSession::SetState: VP XXX2 [OutOfSync -> Connected], state change locked!
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaSession::DoSetContext: setContext for VP XXX2 (url: https://xxx2.xxx.xxx.co:9997/axis2/services/vasa2) failed [connectionState: Connected]: STORAGE_FAULT (ADB conversion error. Details: Invalid Fault Id / )
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] ScheduleEventPollLocked: XXX2 Unable to cancel timer, bailing out.
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaSession::RemoveAllConnections VP (XXX2), purged 0 connections, 0 currently active, new genId (2984) (broadcast wakeup to all threads waiting for free connection)
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaSession::SetState: VP XXX1 [Connected -> TransportError], state change locked!
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaOp::EventPollerCB [#2098433]: ===> Issuing 'getEvents' to VP [XXX1:TransportError (Outstanding 0/4)]
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaOp[#2098433] ===> FINAL FAILURE getEvents, error (INVALID_SESSION / Bad session state (TransportError)) VP (XXX1) Container (XXX1) timeElapsed=10014 msecs (#outstanding 0)
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaSession::EventPollerCB VP XXX1: getEvents failed (INVALID_SESSION, Bad session state (TransportError)) [session state: TransportError]
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] VasaSession::EventPollerCB VP XXX1: connection state changed. Raising alarm to recalculate best VP for all arrays managed by this VP
VVold[2099552]: [Originator@6876 sub=Default OpId=Session] SI:ProcessVpStateChangeEvent: Processing VP State change for XXX1 (newstate TransportError)
VVold[2104990]: [Originator@6876 sub=Default OpId=Session] ScheduleEventPollLocked: XXX1 Unable to cancel timer, bailing out.

Environment

ESXi 7.0.x
ESXi 8.0.x

Cause

All hosts connected to the provider lose their connections and sessions, forcing them to re-establish these connections.

The Vasa Provider may struggle to handle the concurrent session creation requests (setContext) from all hosts. This leads to the VASA Provider going offline from ESXI perspective (since the host cannot establish any sessions with the VP), rendering the virtual volume datastores inaccessible.

Over time, as the number of concurrent session creation requests decreases, the system should eventually recover.

Resolution

Engage storage vendor to ensure the VP handles concurrent connection requests efficiently.