Recovering powered on virtual machines in a device that has encountered a PDL condition

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides an overview of what a Permanent Device Loss (PDL) is and the steps to power on virtual machines that are located on a device that has encountered a Permanent or Transient PDL condition.

Symptoms:

A device with powered on virtual machines has encountered a PDL condition.

Environment

VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.0

Resolution

Permanent PDL is the action that results when you modify the storage array configuration by removing, masking, or unmasking LUNs or by bringing down a path or a component in the path. It may also be caused by storage manager actions such as array lease rotation, rebalancing capacity/workload retiring storage, or maintaining components in the path.

Transient PDL is the action that results due to changes in the storage that are not triggered by administrative actions. These are unusual events and, in general, are not common routines. For example, a dead interface such as Network switch, HBA, or array, or storage network reconfiguration.

To recover powered up virtual machines on a device that has been affected by PDL to the device/LUN:

1. Run this command to list the virtual machines and identify the virtual machines that are to be powered on:
   
   esxcli vm process list
   
   You see an output similar to:
   
   RH4U5-101VM
   World ID: 8339
   Process ID: 0
   VMX Cartel ID: vm_cartel_ID
   
   Where vm_cartel_ID is the cartel ID of the virtual machine to be powered on.
   
   
2. Power off the virtual machine from the vSphere Client or the vSphere Web Client. If the virtual machine is unresponsive, execute the below command to kill or power down the virtual machine:
   
   kill -9 vm_cartel_ID
   
   
3. Run this command to remap or to bring the PDL device online:
   
   esxcli storage core adapter rescan -A vmhbax
   
   For example:
   
   esxcli storage core adapter rescan -A vmhba1
   esxcli storage core adapter rescan --adapter vmhba1
   
   Note: Alternatively, you can also use the VMware vSphere Client to bring the device online. To bring the devices online using the vSphere Client, navigate to Host > Configuration > Storage > Datastores and click Rescan All.

Additional Information

You can also use these steps to list the open files on the datastore you are trying to recover, so that you know the virtual machines or processes to kill:

1. Run this command to obtain the datastore UUID:
   
   esxcfg-scsidevs -m
   
   You see output similar to:
   
   naa.60060160729025007628b54969f4e211:1 /vmfs/devices/disks/naa.60060160729025007628b54969f4e211:1 ########-####-##########7d 0 VMFS5-Datastore1
   
   In this output, ########-####-##########7d is the UUID for datastore VMFS5-Datastore1.
   
   
2. Run this command against the datastore UUID:
   
   lsof | grep -i datastore_UUID
   
   You see output similar to:
   
   1982587 vmx 12 35 /vmfs/volumes/########-####-##########7d/vMotion-test3/vMotion-test3-000001-delta.vmdk
   1982587 vmx 12 36 /vmfs/volumes/########-####-##########7d/vMotion-test3/vMotion-test3-flat.vmdk
   
   The process ID in this output is 1982587.

Note: Killing non-virtual machine processes may have adverse impact on the ESXi host state. If hostd or other ESXi host specifiec processes are listed as holding lock on the datastore, rebooting the ESXi host is a safer option.