Shared Disks Show Zero Capacity After Storage Migration for Microsoft Windows Failover Cluster

Products

• VMware vSphere ESXi

Issue/Introduction

• After migration of VM with shared disks, such as Microsoft Windows Failover Cluster, the guest OS will not bring shared disks online.
  
  
• Windows may report shared disks with zero capacity and fail to bring the disks online.
  
  
• Event logs within the guest operating system indicate that the quorum disk is missing or inaccessible.
  
  
• From an ssh session of the host where the VM is registered, the directory containing the naa.### to vml.## symbolic links show multiple vml.### IDs linked to the same naa:

ls -l /vmfs/devices/disks | grep naa.################################

-rw-------  1  root  root      ############ Aug 10 23:29 naa.################################

-rw-------  1  root  root      ############ Aug 10 23:29 naa.################################:1

lrwxrwxrwx  1  root  root      ## Aug 10 23:29 vml.<ID 1> -> naa.################################

lrwxrwxrwx  1  root  root      ## Aug 10 23:29 vml.<ID 1> -> naa.################################:1

lrwxrwxrwx  1  root  root      ## Aug 10 23:29 vml.<ID 2> -> naa.################################

lrwxrwxrwx  1  root  root      ## Aug 10 23:29 vml.<ID 2> -> naa.################################:1

 

...where <ID 1> and <ID 2> are 2 different hexadecimal strings (these may only vary by one or 2 digits).

 

 

Environment

• VMware vSphere ESXi 8.x

Cause

• The issue is caused by a change on the SAN for the underlying LUN where the LUN ID has changed (and thus the associated vml.ID) without detaching the LUN from the host and remapping the RDM's. Some examples of this are migration of the LUN from one SAN to another or re-presentation of the LUN with new LUN ID's due to host to LUN mapping changes. 
  
  
• When this is done, the ESXi host storage subsystem retains stale vml.### links, resulting in a single naa.### ID being linked to multiple vml.### IDs.
  
  
• Since the RDM mapping refers to the original vml.### ID. This prevents the guest operating system from correctly identifying and mounting the shared RDM devices to the VM's.

Resolution

1. Clear stale mappings:

• Reboot the affected ESXi host to clear the stale LUN vml.### IDs.
  
  

2. Verify which vml ID the VM's RDM references:

cd /vmfs/volumes/<Datastore_name>/<VM_name>

vmkfstools -q VM_name.vmdk

You should see output similar to:

Disk VM_name.vmdk is a Passthrough Raw Device Mapping
Maps to: vml.<ID 1>

 

3. Remove and re-add RDMs, if required:  

• After the reboot, if the vml ID from the above command output does not match the linked vml in the /vmfs/devices/disks directory:
  
  
  ls -l /vmfs/devices/disks | grep naa.################################
  -rw-------  1  root  root      ############ Aug 10 23:29 naa.################################
  -rw-------  1  root  root      ############ Aug 10 23:29 naa.################################:1
  lrwxrwxrwx  1  root  root      ## Aug 10 23:29 vml.<ID 2> -> naa.################################
  lrwxrwxrwx  1  root  root      ## Aug 10 23:29 vml.<ID 2> -> naa.################################:1
  
  
  ...then remove and re-add the RDM(s) to ensure the correct LUN reference.  See KB: Add new RDM to existing Microsoft Windows Server Failover Clustering (WSFC) VMs

Additional Information

• For information on how to unmounting/detach LUN's, see KB: Detach a LUN device from ESXi hosts.
  
  
  
• For supported configurations regarding Microsoft Windows Server Failover Clustering on vSphere, refer to: Microsoft Windows Server Failover Clustering (WSFC) with shared disks on VMware vSphere 8.x: Guidelines for supported configurations