Test Recovery Cleanup fails with error : Cannot unmount datastore ''##########' from host ##########.The resource datastore is in use.

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• The Advanced Setting parameter for storageProvider.datastoreUnmountRetryCount is set = 3, Path :
  SRM UI -> Summary -> Advanced Parameter -> storage Provider -> datastoreUnmountRetryCount.

• The Test Recovery cleanup fails with below error,

  

  Error : "Cannot unmount datastore ''###############################' from host ##########.The resource datastore is in use."

• The error persists even when the 'Force Recovery' option is selected while running Test recovery plan cleanup.

Environment

• VMware Live Recovery 9.0.2

• Dell EMC SRDF

• SRA Version 10.1.0.0

Cause

• The issue stems from the storage array retaining a lock on the datastore created during the Test Recovery plan execution, which keeps the filesystem in a 'busy' state.
• Test Recovery cleanup fails to unmount the datastore because the storage array lock prevents the filesystem from being released. Clearing this lock requires more attempts than the default retry configuration (value=3) allows.

Analysis of /var/log/vmware/srm/vmware-dr.log indicates that the unmount operation failed because the datastore remains in use, specifically due to a busy state on the storage array.

####-##-##T##:##:##.872-05:00 warning vmware-dr[02120] [SRM@6876 sub=HostDomain opID=########-####-####-####-###########-cleanupTest:####:####:####:####] RetryOrFailUnmountVmfsVolumeEx: Batch unmounting of VMFS volume UUID=########-########-####-#####-######## failed with error
--> (vim.fault.ResourceInUse) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.datastore.error.unmountVmfsVolumeFailed",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "name",
-->                value = "Datastore Name: "##############################" VMFS uuid: #########-########-####-############"
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "reason",
-->                value = "fileSystemBusy"
-->             }
-->          ],
-->          message = "Cannot unmount volume 'Datastore Name: "##############################" VMFS uuid: #########-########-####-############' because file system is busy. Correct the problem and retry the operation."
-->       }
-->    ],
-->    type = <unset>,
-->    name = "Datastore Name: ############################## VMFS uuid: #########-########-####-############"
-->    msg = "The resource 'Datastore Name: ############################## VMFS uuid: #########-########-####-############' is in use."
--> }
--> Will retry (attempt 1) in 1 seconds

Resolution

Below steps performed to resolve the issue,

1. Open SRM UI ->Summary -> Advanced Parameter -> storage Provider.
2. The parameter : storageProvider.datastoreUnmountRetryCount default value will be set to 3 (Default value), increase the value to 5.
3. Save the value and perform this step on both sites.
4. Followed by Re-running the cleanup.

Additional Information

For reference, Configuring advanced VMware SRM options