The snapshot fails to revert, causing the VM to remain powered off and potentially affecting VM accessibility
search cancel

The snapshot fails to revert, causing the VM to remain powered off and potentially affecting VM accessibility

book

Article ID: 387820

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

The snapshot fails to revert, causing the VM to remain powered off and potentially affecting VM files. After some time, the same snapshot reverts without issues. 

Test 1:

  • Snapshot creation and power cycling of the VM completed without issues.
  • Snapshot revert attempts triggered errors: "An error occurred while reverting to a snapshot: 16 (Device or resource busy)."
  • Reverting the same snapshot succeeded after a delay of six days.

Test 2:

  • Snapshot creation, revert attempts, and power cycling were initially successful.
  • On the third revert attempt, the same error occurred as in Test 1.

Key Observations:

  • Snapshot creation on VVOLs succeeds consistently but fails unpredictably during revert operations. These failures leave the VM powered off, potentially impacting VM files.
  • After a delay, the same snapshot reverts without errors, indicating the issue is sporadic, not persistent.

Verification Steps: 

  • Host is able to access Protocol End Point (PE)

naa.60060#######################FC10:
Host Id: naa.60060#######################fc10
Array Id: Hitachi:<Array_ID>
Type: SCSI
Accessible: true
Configured: true
Lun Id: naa.60060#######################fc10
Remote Host:
Remote Share:
NFS4x Transport IPs:
Server Scope:
Server Major:
Auth:
User:
Storage Containers: ########-####-####-a26c-############

  • Host is able to access storage container

H2-T0-Coral-VVOL:
StorageContainer Name: Example-VVOL
UUID: vvol:a1##########4a00-a2##########1d2b
Array: Hitachi:<Array_ID>
Size(MB): 413288601
Free(MB): 167610315
Accessible: true
Default Policy:

  • VASA communication is established and VASA status is online

Hitachi VIC VASA:VP
Name: Hitachi VIC VASA
URL: Status: online
Arrays:
Arrays:
ArrayId: Hitachi:<Array_ID>
Active: true
Priority: 0

Environment

vSphere ESXi 7.x 
vSphere ESXi 8.x 

vSphere vVOL

Cause

The busy signals originate from the storage as the host interacts with the VASA provider to retrieve information about the VVOL volumes. This behavior does not necessarily point to a hardware fault in the storage. Instead, it suggests a resource constraint or increased load on the VASA provider or the storage system.

Validation Steps: 

  •  The revert operation failed due to the Device or resource busy

info vpxa[2102124] [Originator@6876 sub=Default opID=m3qxojly-767181-auto-gfym-h5:70050072-d-c5] [VpxLRO] -- ERROR task-124999 -- vpxa -- vpxapi.VpxaService.revertToSnapshot: vim.fault.GenericVmConfigFault:
--> Result:
-->          key = "msg.snapshot.vigor.revert.error",
-->                value = "16 (Device or resource busy)"
-->          message = "An error occurred while reverting to a snapshot: 16 (Device or resource busy)."
-->    reason = "An error occurred while reverting to a snapshot: 16 (Device or resource busy)."
-->    msg = "An error occurred while reverting to a snapshot: 16 (Device or resource busy)."

  • The task encountered an error: VVolLib_IpcStorageFaultToVVolLibError:766: Storage Fault RESOURCE_BUSY (14). The VASA task failed with the status: FAILURE: (RESOURCE_BUSY). This indicates that the resource in question is currently busy, causing the task to fail. 

2024-12-13T03:37:00.091Z info hostd[2101138] [Originator@6876 sub=Libs opID=m3qxojly-767181-auto-gfym-h5:70050072-d-c5-6e89 user=vpxuser:VPX.User] 2101138:VVOLLIB : VVolLib_IpcStorageFaultToVVolLibError:766: Storage Fault RESOURCE_BUSY (14):
2024-12-13T03:37:00.091Z error hostd[2101138] [Originator@6876 sub=Libs opID=m3qxojly-767181-auto-gfym-h5:70050072-d-c5-6e89 user=vpxuser:VPX.User] 2101138:VVOLLIB : VVolLibProcessTaskUpdate:2334: VASA task (arrayID:'<Array_ID>', task:'########-####-####-a6d1-############') has completed with FAILURE: (RESOURCE_BUSY, ) The VVol-related resource is busy and cannot be used

  • OSFSD encountered a storage fault with the error NOT_FOUND (13): An empty response was returned, indicating that no VVOL was found for the specified VVOL ID

osfsd.log:2024-12-13T03:31:57.711Z info osfsd[29450029] [Originator@6876 sub=Vvol] VVolMount@2894:Lookup was sent friendly name catalog
osfsd.log:2024-12-13T03:31:57.740Z info osfsd[29450029] [Originator@6876] 29450029:VVOLLIB : VVolLib_IpcStorageFaultToVVolLibError:766: Storage Fault NOT_FOUND (13): Empty response
osfsd.log:2024-12-13T03:31:57.740Z warning osfsd[29450029] [Originator@6876] 29450029:VVOLLIB : VVolLib_FriendlyNameToUUID:4969: QueryUUIDByName (catalog) Failed with not found

  • VVOLD encountered an issue and was unable to retrieve information about the volumes. This indicates a failure in accessing or querying the required volume. 

    2024-12-13T03:37:01.609Z info vvold[20959517] [Originator@6876 sub=Default opID=m3qxojly-767181-auto-gfym-h5:70050072-d-c5-6e89] VvolServiceInstance::GetVirtualVolumeTaskInfoFromVasa estimatedSecs = 0
    2024-12-13T03:37:01.609Z info vvold[20959517] [Originator@6876 sub=Default opID=m3qxojly-767181-auto-gfym-h5:70050072-d-c5-6e89] SI:GetTaskUpdate success for taskId: ########-####-####-b561-############
    2024-12-13T03:37:01.609Z info vvold[20959517] [Originator@6876 sub=Default opID=m3qxojly-767181-auto-gfym-h5:70050072-d-c5-6e89] VASA task revertVirtualVolume (id: ########-####-####-b561-############) FAILED in 1516 ms (5 retries), not updating poll interval

Resolution

The issue requires investigation by the storage vendor in order to get this addressed. 

 

Additional Information

Note: In this case, the storage vendor has recommended upgrading the storage firmware to address the issue. 

Powered by Wolken Software