On LUN failover, ESXi host is destabilized by LUN write-protected warnings

Products

• VMware vSphere ESXi

Issue/Introduction

•  An environment is configured with array replication of LUNs to a DR site
  
  
• A failover of a number of LUNs is completed to the DR site (i.e. production VMs shut down and unregistered, production LUNs demoted, DR LUN promoted, VMs regisered powered up on DR site)
  
  
• After failover, one production ESXI becomes less reponsive/unreponsive 
  
  
• A large volume of LUN write-protected SCSI warnings (H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x27 0x0) are reported in the host /var/log/vmkernel.log files.

Environment

VMware vSphere ESXi (all versions)

Cause

The datastores are not unmounted and the LUN are not detached before failover.

While all production VMs are shutdown and unregistered a vCLS VM remains running on one of the LUNs. This vCLS VM repeatedly attempts to process I/O against the datastore whose backing LUN is write protected, triggering the large volume of SCSI Write-Protection errors.

Resolution

To resolve when the issue has occured:

1. Configure the target deployment datastore for vCLS VMs to one that is not write-protected. See  External vCLS Datastore Placement
   
   
2. Enable retreat mode (this should power off and delete the vCLS VM) Disable vCLS on a Cluster via Retreat Mode
   
   
3. If previous step successfully deletes the vCLS VM, disable retreat mode (vCLS VMs will then be created only on the datastore or datastore(s) specified in Step 1)
   
   
4. If vCLS VM fails to delete, power down host '0905'. Then, disable retreat mode. Power host '0905' back on. The old vCLS VM registration on '0905' will need to be cleaned up. 

 

To avoid this issue:

1. Configure a set of datastores which can be used for the vCLS VMs, which won't be subject to the failover process. See External vCLS Datastore Placement
   
   
2. More generally, it is good practice to unmount a datastore from hosts and the detach the LUN before failover. ESXi will then treat the LUN as administratively turned off. Detach a LUN device from ESXi hosts

Additional Information

See Datastore hosting vCLS VMs does not enter maintenance mode