Troubleshooting LUNs detected as snapshot LUNs in vSphere
search cancel

Troubleshooting LUNs detected as snapshot LUNs in vSphere

book

Article ID: 323142

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

This article provides information on how snapshot LUN detection is handled by ESXi.

Symptoms:

  • A datastore is missing, though you can still see the LUN presented to the host
  • In the vmkernel.log file, you see entries similar to:
# Example 1:
LVM: 8445: Device naa.xxxxxxxxxxxxxxxxxxxx:1 detected to be a snapshot:
LVM: 8445: Device eui.xxxxxxxxxxxxxxxxxxxx:1 detected to be a snapshot:
LVM: 8452: queried disk ID: <type 1, len 17, lun 36, devType 0, scsi 0, h(id) 7683208289187576905>
LVM: 8459: on-disk disk ID: <type 1, len 17, lun 17, devType 0, scsi 0, h(id) 7683208289187576905>

#Example 2:
LVM: 11764: Device naa.xxxxxxxxxxxxxxxxxxxx:1 detected to be a snapshot:
LVM: 11770:   queried disk ID: <type 2, len 22, lun 0, devType 0, scsi 0, h(id) 15436777671286394200>
LVM: 11777:   on-disk disk ID: <type 2, len 22, lun 0, devType 0, scsi 0, h(id) 16841058529903986994>

 

  • When forcing a LUN that is formatted as VMFS-5 or VMFS-6 to mount (either with the name or UUID), you see entries similar to:
esxcli storage vmfs snapshot mount -l LUN
No unresolved VMFS snapshots with volume label 'DEV-LUN' found.

esxcli storage vmfs snapshot mount -u UUID
No unresolved VMFS snapshots with volume UUID '4f1d6367-########-####-########149a' found.
 
  • Force mounting a VMFS datastore results in these symptoms:
    • Other hosts in the same datacenter cannot mount that VMFS datastore from vCenter Server.
    • The Resolve VMFS Volumes automatic task, seen in the task list of the vCenter Client, results in the entries similar to:
Error: Cannot change the host configuration.
Error Stack
Call "HostStorageSystem.ResolveMultipleUnresolvedVmfsVolumes" for object "storageVolume" on vCenter Server "VC" failed.

Cause

The snapshot LUNs issue occurs when the ESX host does not confirm the identity of the LUN with what it expects to see in the VMFS metadata.
 
This issue occurs after replacing SAN hardware, firmware upgrades, SAN replication, DR tests, and some HBA firmware upgrades.


Force mounting a VMFS datastore fails if:
  • Multiple ESXi hosts are managed by the same vCenter Server and these hosts are in the same datacenter.
  • A snapshot LUN containing a VMFS datastore is presented to all these ESXi hosts.
  • One of these ESXi hosts force mounts the VMFS datastore that resides on this snapshot LUN.
  • A second ESXi host is attempting to do an operation at the same time.
When one ESXi host force mounts a VMFS datastore residing on a LUN is detected as a snapshot, an object is added to the datacenter grouping in the vCenter Server database to represent that datastore.

When a second ESXi host attempts to do the same operation on the same VMFS datastore, the operation fails because an object already exists within the same datacenter grouping in the vCenter Server database.

Because an object already exists, vCenter Server does not allow mounting the datastore on any other ESXi host residing in that same datacenter.

Resolution

Important Notes :

  • Keeping the existing signature will cause problems in the future.
    • Such as preventing the increase of the datastore size, and preventing the datastore from automatically mounting on new hosts, etc. So, "force mounting" (mounting the datastore while keeping the existing signature) is just a temporary step to gain immediate access.  

 

    • After which, please either schedule a maintenance window for the VMs on that datastore and assign the new signature, or migrate the VMs off the datastore so they are not affected by the maintenance.

 

  • On resignature, in addition to assigning a new UUID, ESXi assigns a modified label (i.e. name) to the datastore. The default format of the new label assigned to the datastore is snap-snapID-oldLabel, where snapID is an integer and oldLabel is the label/name of the original datastore.

 

ESXi 7.x, and 8.x

vSphere Client

  1. Log in to the vSphere Client and navigate to vCenter Home.
  2. Click Datastores in the menu on the left.
  3. In the Objects tab click the Create a new datastore icon in the top left.
  4. Type the datastore name and if required, select the placement location for the datastore.
  5. Select VMFS as the datastore type.
  6. From the list of storage devices, select the device that has a specific value displayed in the Snapshot Volume column.

    Note: The value present in the Snapshot Volume column indicates that the device is a copy that contains a copy of an existing VMFS datastore.
  7. Under Mount Options, select the desired option (Keep existing, Assign new, Format) for your volume and click Next.
  8. Review the datastore configuration information.
  9. Click Finish.


Command line

  1. Connect to the ESXi host with an SSH session and root credentials
  2. To list the volumes detected as snapshots, run this command:
esxcli storage vmfs snapshot list

You see output similar to:
xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx
Volume Name: VMFS_1
VMFS UUID: xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx
Can mount: true
Reason for un-mountability:
Can resignature: true
Reason for non-resignaturability:
Unresolved Extent Count: 1
  1. To resignature a snapshot/replica LUN (the volume is mounted immediately after the resignature with a new UUID,), run this command:

Please note the steps and warnings outlined in the Additional Information before executing this command.

esxcli storage vmfs snapshot resignature -l label|-u uuid

For example:
esxcli storage vmfs snapshot resignature -l "VMFS_1"
esxcli storage vmfs snapshot resignature -u "xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx"
  1. To (persistently force) mount a snapshot/replica LUN across reboots, run this command:
esxcli storage vmfs snapshot mount -l label|-u uuid
  1. To (noN persistent force) mount a snapshot/replica LUN that is not persistent across reboots, run this command:
esxcli storage vmfs snapshot mount -n -l label|-u uuid
  1. To mount the volume without performing a resignaturing of that volume (this volume is mounted when the ESX host is rebooted), run this command:
esxcfg-volume -M VMFS_UUID|label

For example:
esxcfg-volume -M "VMFS_1"
esxcfg-volume -M "xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx"

Note: To view the datastores again in vCenter Server, you may have to perform a rescan of the storage adapters on all ESXi hosts that the datastore is presented to or a refresh of the storage view. If you are having trouble identifying the affected datastore, in the vSphere client, check the storage view of another ESXi host that still has the datastore mounted correctly. This will then allow you to correlate VMFS datastore name with NAA LUN identifier.

Note: Attempting to resignature an already mounted snapshot LUN fails: see fault.VmfsAlreadyMounted error when resignaturing a snapshot LUN (broadcom.com) 

 

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ (broadcom.com)
 


Best practices for mounting snapshot LUNs

#Note:  Do not resignature datastores that have running VMs/VMDKs, as In the process of the resignature the UUID/Path to the vms and disks will change will lead to an outage.

# This step must be carried out while vms are shutdown, unregistered and the datastore is unmounted from all ESXi hosts.

  1. Hence there must be no active I/O on the affected datastore(s). Any registered virtual machines on the volume prevent the datastore from being unmounted and the host database location record will be incorrect after the datastore is resignatured and the virtual machine within will not power on.
    You have two options to ensure no I/O occurs on the datastore:
     
      1. Storage vMotion: Either migrate all virtual machines off the datastore using Storage vMotion (from any and all hosts with) to another datastore by placing datastore into maintenance mode.
        1. At which point you may wish to consider presenting a new datastore and removing the snapshot lun limiting any downtime.

          OR
           
      2. Shutdown and unregister: Shutdown and unregister any virtual machines that reside on the affected datastore.
         
    2. Unmount the datastore on all hosts.
    3. Select the LUN with your VMFS volume data on it and add the storage to one host in the cluster with the Assign a New Signature option.

      Note:
      • The snapshot datastore can be mounted to one host only until it is resignatured.
      • This does not have to be host you have problems mounting it to.
      • Rename the volume if required. By default, the datastore name appends with snap- after the resignature operation.
         
    4. Add any virtual machines from the resignatured datastore back to the inventory and power them on.

      Virtual machines which primarily reside on a different datastore but partly reside on the resignatured volume (they have one or more disks on the affected datastore) may need to have the virtual disk (vmdk) removed from the virtual machine configuration and re-added from the newly resignatured datastore location.

 

Important notes to keep in mind when force mounting a VMFS volume on an ESXi host:

  • A VMFS datastore is mounted only if it does not collide with an already mounted VMFS datastore that has the same UUID (that is, VMFS signature). In other words, if the original LUN that contains the original VMFS datastore together with its snapshot LUN are both presented to the ESXi host simultaneously, force mounting the VMFS datastore from the snapshot LUN on this same ESXi host is not allowed. In this case, the only way to mount that VMFS datastore from the snapshot LUN is to resignature and choose a new label for that datastore.

 

  • In a datacenter that has multiple ESXi hosts, a VMFS datastore on a snapshot LUN is force mounted on any one of these hosts. When this datastore is mounted on a given host, that datastore cannot be force mounted on any other host using the vCenter Server user interface. To mount that datastore on other hosts, use the preceding steps.

 

  • If the datastore will not mount correctly in vCenter Server, retry the mount while directly connected to the ESX host. Other hosts see the volume after the resignature operation. If you do not see the volume after the resignature operation, do a manual rescan from each host(s) as required, see Performing a rescan of the storage on an ESXi host (broadcom.com)

 

Identify if a datastore is force mounted on an ESXi host (broadcom.com)

Extending or increasing a datastore through vCenter Server fails (broadcom.com)

 

Powered by Wolken Software