VMs recovered using Dell NetWorker fails to power ON due to missing disk

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• VM's vmdk missing after recovery.

• When the VM's folder is validated the all the recovered vmdk files are not present, even when NetWorker reports that the recovery was successful.

Environment

VMware ESXi 8.x

VMware vCenter 8.x

Cause

• If DRS/SDRS initiates migration of the backup proxy VM, during the timeframe of the recovery task, the vmdks of the VM being recovered will remain attached to the proxy VM. (During recovery process, the recovered vmdks are initially attached to the backup proxy VM for recovery.)

• This is because during VM migration (compute or storage), the VM takes ownership of the disks attached to it, here the backup proxy VM.

• Now the recovery task may complete successfully, but the vmdk will remain attached to the proxy VM and migrated to it's folder, and the recovered VM would be missing it's vmdk(s).

• Below is an example of what this issue looks like from the logs:

  1. From vCenter's /var/log/vmware/vpxd/vpxd.log it can be first seen that a recovered vmdk gets created and attached to the recovered VM:
     
     YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[05965] [Originator@6876 sub=VmProv opID=########-01] Applying ConfigSpec (vim.vm.ConfigSpec) {
.
-->          operation = "add",
-->          fileOperation = "create",
-->          device = (vim.vm.device.VirtualDisk) {
.
.
-->                fileName = "ds:///vmfs/volumes/########-########-####-############/Recovered_VM/Recovered_VM_1.vmdk",
.
.
--> } to VM ds:///vmfs/volumes/########-########-####-############/Recovered_VM/Recovered_VM.vmx
on host ##.##.##.##
     
  2. Then after a few seconds, the same vmdk gets attached to a backup proxy VM for data recovery: 

     YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[06635] [Originator@6876 sub=VmProv opID=#######-01] Applying ConfigSpec (vim.vm.ConfigSpec) {
.
-->          operation = "add",
.
-->                fileName = "ds:///vmfs/volumes/########-########-####-############/Recovered_VM/Recovered_VM_1.vmdk",
.
--> } to VM ds:///vmfs/volumes/########-########-####-############/Backup_Proxy_VM/Backup_Proxy_VM.vmx on host ##.##.##.##

  3. A few seconds after this, a migration task can be seen initiated for the Backup_Proxy_VM VM from host-01 to host-02 for DRS. Similarly for SRDS in the below log snippet, the ds (datastore) location would be different:
     

     YYYY-MM-DDTHH:MM:SS.SSSZ info vpxd[06199] [Originator@6876 sub=VmProv opID=lro-1-########-###-01-01-01] Local-VC Datastore Migrate of poweredOn VM 'Backup_Proxy_VM' (vm-###, ds:///vmfs/volumes/########-########-####-############/Backup_Proxy_VM/Backup_Proxy_VM.vmx) on host-01 (##.##.##.##) in pool resgroup-## with ds ds:///vmfs/volumes/########-########-####-############/ to host-02 (##.##.##.##) in pool resgroup-## with ds ds:///vmfs/volumes/########-########-####-############/

  4. Once the recovery task is completed successfully, if the recovered VM is validated, it would be missing its vmdks and if the backup proxy VM is validated, it would have the recovered vmdks still attached to it.

Resolution

To resolve this issue, the below points should be followed as per Dell recommendation:

1. DRS should be set to manual for the backup proxy VMs:

1. On the vSphere Client, select the vSphere Cluster where the backup proxy VM resides.

2. Select Configure > Configuration - VM Overrides.

3. Click on "Add" and in the window that pops up, select the backup proxy VMs, and click "Next".

4. Click on the "Override" checkbox for vSphere DRS - DRS automation level and select "Disabled".

5. Click on "Finish" to save the configuration.

2. SDRS should also be set to manual for the backup proxy VMs

1. On the vSphere Client, select the datastore cluster where the backup proxy VM resides.

2. Select Configure > Configuration - VM Overrides.

3. Click on "Add" and in the window that pops up, select the backup proxy VMs, and click "Next".

4. Click on the "Override" checkbox for the Storage DRS Automation Level and select "Disabled".

5. Click on "Finish" to save the configuration.

Additional Information

Reference document from Dell for this issue - NetWorker NVP-vProxy: Image recovery is marked as successful but the VM cannot start due to missing disk.