Possible Migration failures of virtual machines stored on vVols due to overloaded VASA providers

Products

• VMware vSphere ESXi

Issue/Introduction

When initiating many vMotion operations from a single host (manually or via entering Maintenance Mode), you observe that not all vMotions are successful and error out with a generic error: "A system error occurred" .

Upon review of vvold.log you observe VM swap file creation taking a long time and/or the number of connections to the VASA Provider max out:
Note: vvold.log is located in /var/run/log/vvold.log

Long creation time for swap:
[YYYY-MM-DDTHH:MM:SS] error vvold[2104266] [Originator@6876 sub=Default]
--> VasaOp::ThrowFromSessionError [#129228]: ===> FINAL SUCCESS createVirtualVolume VP (##########-ct0) Container (####.########-####-####-####-############) timeElapsed=17814 msecs (#outstanding 4)

VASA Provider (VP) connections exhausted:
[YYYY-MM-DDTHH:MM:SS] info vvold[2099736] [Originator@6876 sub=Default] Came to SI::BindVirtualVolume: esxContainerId ####.########-####-####-####-############ VVol Id rfc####.########-####-####-####-############ bindType Normal (isConfigVvol: false)

[YYYY-MM-DDTHH:MM:SS] info vvold[2099736] [Originator@6876 sub=Default]
--> VasaOp::BindVirtualVolume [#129239]: ===> Issuing 'bindVirtualVolume' to VP ##########-ct0 (#outstanding 5/5) [session state: Connected]

[YYYY-MM-DDTHH:MM:SS] info vvold[2099736] [Originator@6876 sub=Default] VasaSession::GetFreeConn: No free connections to VP (##########-ct0), numConns (5)!

 

[YYYY-MM-DDTHH:MM:SS] Er(163) VVold[2886804]: [Originator@6876 sub=Default IpcCtx=2100018-16 OpId=nbdmode-000000##########] VasaOp::IsSuccessful [#565991]: bindVirtualVolume transient failure: 15 (PROVIDER_BUSY / No free vasa connection) [internal busy]

[YYYY-MM-DDTHH:MM:SS] Wa(164) VVold[2886804]: [Originator@6876 sub=Default IpcCtx=2100018-16 OpId=nbdmode-000000##########] VasaOp[#565991] ===> Transient failure bindVirtualVolume VP (XXXXX-ct1) retry=true, batchOp=false container=####.########-####-####-#### timeElapsed=0 msecs (#outstanding 5)

[YYYY-MM-DDTHH:MM:SS] In(166) VVold[2886804]: [Originator@6876 sub=Default IpcCtx=2100018-16 OpId=nbdmode-000000##########] VasaOp::TimedOut: [#565991] No free connection to VP (XXXX-ct1) for Op (bindVirtualVolume), waiting for free connection, from max (5) connections [89999 msecs to timeout]

 

Environment

VMware vSphere ESXi 6.X
VMware vSphere ESXi 7.X
VMware vSphere ESXi 8.X

Resolution

  Ask your storage partner if moving to a more recent vasa provider version would help resolve this issue. Also check load/performance  on the backend storage array.