Backups are failing with error "The vVol operation timed out"

Products

Issue/Introduction

Symptoms:

• Backup job fails intermittently with an error messages as:

  • "An error occurred while saving snapshot, the vVol VASA provider returned an incomplete response. unable to save snapshot. The vVol operation timed out"

Validation:
From /vmfs/volumes/<datastore_name>/<vm_name>/vmware.log, can observe an event for "VVol operation timeout"

2025-04-12T19:39:32.117Z In (05) vcpu-0 - 8338224:VVOLLIB : VVolLib_IpcStorageFaultToVVolLibError: 766: Storage Fault TIMEOUT (19) : TIMEOUT: Retry Timeout
2025-04-12T19:39:32.117Z Er (02) vcpu-0 - 8338224:VVOLLIB : VVolLib_VectoredSnapshotVVol: 6802: Failed with error  (22) The VVol operation timed out
2025-04-12T19:39:32.117Z Er (02) vcpu-0 - 8338224:VVOLLIB : VVolLib_VectoredSnapshotVVol:6806: Vectored VvolSnapshot failed: The Vol operation timed out.
2025-04-12T19:39:32.117Z Er (02) vcpu-0 - OBJLIB-VVOLOBJ : VVolObjNotifySnapshot: Could not take vectored snapshot.
2025-04-12T19:39:32.117Z In(05) vcpu-0 - SNAPSHOT :: SnapshotFinishCreate: ObjLib_NotifySnapshot failed: The VVol operation timed out

Environment

VMware ESXi 7.x

Cause

The error is caused by a TRANSPORT FAULT timeout, which occurs when the VASA API does not receive a response from the client side within the 30-second timeout window, leading to the failure of the API call.

From ESXi,
var/run/log/vvold.log there is an error reported as "Transport fault timeout"

2025-04-09T19:15:51.560Z warning vvold[2578431] [Originator@6876 sub=Default] VasaOp[#2826826] ===> Transient failure snapshotVirtualVolume VP (###########) retry=true, batchOp=true container=87842230-####-####-####-77fa3a085f61 timeElapsed=30994 msecs (#outstanding 6)
2025-04-09T19:15:51.560Z info vvold[2578431] [Originator@6876 sub=Default] VasaOp::DoRetry [#2826826]: snapshotVirtualVolume got connection-local error (TransportError) from VP (UGPSANSTORP04), will retry on another connection from the same vasa session
2025-04-09T19:15:51.560Z error vvold[2578431] [Originator@6876 sub=Default] VasaOp::TimedOut [#2826826]: ===> snapshotVirtualVolume failed: (TRANSPORT_FAULT /UNKNOWN FAULT) - RETRY TIMEOUT (30/30 secs)!
2025-04-09T19:15:51.560Z error vvold[2578431] [Originator@6876 sub=Default] VasaOp[#2826826] ===> FINAL FAILURE snapshotVirtualVolume, error (TIMEOUT / Retry Timeout) VP (UGPSANSTORP04) Container (87842230-####-####-####-77fa3a085f61) timeElapsed=30994 msecs (#outstanding 6)

Resolution

 Recommend to reach out to the hardware vendor to investigate on timeout error.