Troubleshooting LUNs detected as snapshot LUNs in vSphere
search cancel

Troubleshooting LUNs detected as snapshot LUNs in vSphere

book

Article ID: 323142

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

Symptoms:

  • A datastore was accidentally removed from the hosts and from the SAN. The LUNs were re-created on the SAN and then were re-attached but the storage is available inconsistently on the hosts.

  • A datastore is missing, though the LUN is still presented to the host.

  • One host can see the datastore while other hosts only see the LUN and not the datastore residing on it.

  • After presenting a LUN to an ESXI host, the ESXI host does not mount the datastore.

    Note : The LUN has mounted datastore on other ESXI hosts. The LUN is available with active path on all the hosts.
  • When forcing a LUN that is formatted as VMFS-5 or VMFS-6 to mount (either with the name or UUID), you see entries similar to:

    # esxcli storage vmfs snapshot mount -l <LUN_name>
    No unresolved VMFS snapshots with volume label 'LUN_name' found.

    # esxcli storage vmfs snapshot mount -u UUID
    No unresolved VMFS snapshots with volume UUID '########-########-####-########' found.

  • Force mounting a VMFS datastore results in the following:

    • Other hosts in the same datacenter cannot mount that VMFS datastore from vCenter Server.

    • The Resolve VMFS Volumes automatic task, seen in the task list of the vCenter Client, results in the entries similar to:
Error: Cannot change the host configuration.
Error Stack
Call "HostStorageSystem.ResolveMultipleUnresolvedVmfsVolumes" for object "storageVolume" on vCenter Server "VC" failed.
  • Storage devices corresponding to these LUNs show-up as 'Not consumed' in vSphere Client.

Environment

VMware vSphere ESXi (All Versions)

Cause

  • The snapshot LUN issue occurs when the ESXi host does not confirm the identity of the LUN with what it expects to see in the VMFS metadata.

  • This issue occurs after replacing SAN hardware, firmware upgrades, SAN replication, DR tests, and some HBA firmware upgrades. This can also happen when the LUN ID mapped to a host does not match the LUN ID originally mapped to the host which created the VMFS datastore (as LUN ID is part of the device target path, and so is part of the VMFS metadata).

  • Force mounting a VMFS datastore fails if:

    • Multiple ESXi hosts are managed by the same vCenter Server and these hosts are in the same datacenter.

    • A snapshot LUN containing a VMFS datastore is presented to all these ESXi hosts.

    • One of these ESXi hosts force mounts the VMFS datastore that resides on this snapshot LUN.

    • A second ESXi host is attempting to do an operation at the same time.

  • When one ESXi host force mounts a VMFS datastore residing on a LUN is detected as a snapshot, an object is added to the datacenter grouping in the vCenter Server database to represent that datastore.

  • When a second ESXi host attempts to do the same operation on the same VMFS datastore, the operation fails because an object already exists within the same datacenter grouping in the vCenter Server database, vCenter Server does not allow mounting the datastore on any other ESXi host residing in that same datacenter.

  • In "/var/run/log/vmkernel.log" , you will see below events: 

    # Example 1:
    LVM: 8445: Device naa.####################:1 detected to be a snapshot:
    LVM: 8452: queried disk ID: <type 1, len 17, lun 36, devType 0, scsi 0, h(id) 7683208289187576905>
    LVM: 8459: on-disk disk ID: <type 1, len 17, lun 17, devType 0, scsi 0, h(id) 7683208289187576905>

    #Example 2:
    LVM: 11764: Device naa.####################:1 detected to be a snapshot:
    LVM: 11770:   queried disk ID: <type 2, len 22, lun 0, devType 0, scsi 0, h(id) 15436777671286394200>
    LVM: 11777:   on-disk disk ID: <type 2, len 22, lun 0, devType 0, scsi 0, h(id) 16841058529903986994>

    Note:

      • In Example 1, LUN was originally presented to a host as LUN 17. It was then unmapped and remapped (or presented to a different cluster) using LUN 36.

      • In Example 2, The "h(id)" (Hardware ID/Hash) has changed completely. This usually happens during Storage Replication or a Storage Array Migration. The data was copied from one physical serial number/WWN to a completely different one.

Resolution

Important Notes :

  • Keeping the existing signature will cause problems in the future.

    • Such as preventing the increase of the datastore size, and preventing the datastore from automatically mounting on new hosts, etc. So, "force mounting" (mounting the datastore while keeping the existing signature) is just a temporary step to gain immediate access.  
    • After which, please either schedule a maintenance window for the VMs on that datastore and assign the new signature, or migrate the VMs off the datastore so they are not affected by the maintenance.
  • On resignature, in addition to assigning a new UUID, ESXi assigns a modified label (i.e. name) to the datastore. The default format of the new label assigned to the datastore is snap-snapID-oldLabel, where snapID is an integer and oldLabel is the label/name of the original datastore.

 

 

ESXi 7.x, and 8.x

vSphere Client

  1. Log in to the vSphere Client and navigate to vCenter Home.

  2. Click Datastores in the menu on the left.

  3. In the Objects tab click the Create a new datastore icon in the top left.

  4. Type the datastore name and if required, select the placement location for the datastore.

  5. Select VMFS as the datastore type.

  6. From the list of storage devices, select the device that has a specific value displayed in the Snapshot Volume column.

    Note: The value present in the Snapshot Volume column indicates that the device is a copy that contains a copy of an existing VMFS datastore.

  7. Under Mount Options, select the desired option (Keep existing, Assign new, Format) for your volume and click Next.

  8. Review the datastore configuration information.

  9. Click Finish.

 

 

Command line

  1. Connect to the ESXi host with an SSH session and root credentials.

  2. To list the volumes detected as snapshots, run this command:

$ esxcli storage vmfs snapshot list


You see output similar to:

########-########-####-############


Volume Name: VMFS_1
VMFS UUID:########-########-####-############
Can mount: true
Reason for un-mountability:
Can resignature: true
Reason for non-resignaturability:
Unresolved Extent Count: 1

  1. To resignature a snapshot/replica LUN (the volume is mounted immediately after the resignature with a new UUID,), run this command:

Please note the steps and warnings outlined in the Additional Information before executing this command.

$ esxcli storage vmfs snapshot resignature -l label|-u uuid


For example:

$ esxcli storage vmfs snapshot resignature -l "VMFS_1"

$esxcli storage vmfs snapshot resignature -u "########-########-####-############"

  1. To (persistently force) mount a snapshot/replica LUN across reboots, run this command:

$ esxcli storage vmfs snapshot mount -l label|-u uuid

  1. To (noN persistent force) mount a snapshot/replica LUN that is not persistent across reboots, run this command:

$ esxcli storage vmfs snapshot mount -n -l label|-u uuid

  1. To mount the volume without performing a resignaturing of that volume (this volume is mounted when the ESX host is rebooted), run this command:

$ esxcfg-volume -M VMFS_UUID|label


For example:

$ esxcfg-volume -M "VMFS_1"

$ esxcfg-volume -M "########-########-####-############"

Note:

  • To view the datastores again in vCenter Server, you may have to perform a rescan of the storage adapters on all ESXi hosts that the datastore is presented to or a refresh of the storage view. If you are having trouble identifying the affected datastore, in the vSphere client, check the storage view of another ESXi host that still has the datastore mounted correctly. This will then allow you to correlate VMFS datastore name with NAA LUN identifier.

  • Attempting to resignature an already mounted snapshot LUN fails: see fault.VmfsAlreadyMounted error when resignaturing a snapshot LUN (broadcom.com) 

  • We can also run "esxcfg-volume" cmd to list/mount/resignature the volumes detected as snapshots with few other options given below :
esxcfg-volume

esxcfg-volume <options>
-l|--list                                List all volumes which have been
                                         detected as snapshots/replicas.
-m|--mount <VMFS UUID|label>             Mount a snapshot/replica volume, if
                                         its original copy is not online.
-u|--umount <VMFS UUID|label>            Umount a snapshot/replica volume.
-r|--resignature <VMFS UUID|label>       Resignature a snapshot/replica volume.
-M|--persistent-mount <VMFS UUID|label>  Mount a snapshot/replica volume
                                         persistently, if its original copy is
                                         not online.
-U|--upgrade <VMFS UUID|label>           Upgrade a VMFS3 volume to VMFS5.
-h|--help                                Show this message.




Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ (broadcom.com)

Best practices for mounting snapshot LUNs

Note: 

Do not resignature datastores that have running VMs/VMDKs, as In the process of the resignature the UUID/Path to the vms and disks will change will lead to an outage.

This step must be carried out while vms are shutdown, unregistered and the datastore is unmounted from all ESXi hosts.

  • Hence there must be no active I/O on the affected datastore(s). Any registered virtual machines on the volume prevent the datastore from being unmounted and the host database location record will be incorrect after the datastore is resignatured and the virtual machine within will not power on. You have two options to ensure no I/O occurs on the datastore:
      1. Storage vMotion: Either migrate all virtual machines off the datastore using Storage vMotion (from any and all hosts with) to another datastore by placing datastore into maintenance mode.
        1. At which point you may wish to consider presenting a new datastore and removing the snapshot lun limiting any downtime.

          OR
           
      2. Shutdown and unregister: Shutdown and unregister any virtual machines that reside on the affected datastore. 
  • Unmount the datastore on all hosts.

  • Select the LUN with your VMFS volume data on it and add the storage to one host in the cluster with the Assign a New Signature option.

    Note:
      • The snapshot datastore can be mounted to one host only until it is resignatured.

      • This does not have to be host you have problems mounting it to.

      • Rename the volume if required. By default, the datastore name appends with snap- after the resignature operation.
         
  • Add any virtual machines from the resignatured datastore back to the inventory and power them on.

    Virtual machines which primarily reside on a different datastore but partly reside on the resignatured volume (they have one or more disks on the affected datastore) may need to have the virtual disk (vmdk) removed from the virtual machine configuration and re-added from the newly resignatured datastore location.

Important notes to keep in mind when force mounting a VMFS volume on an ESXi host:

  • A VMFS datastore is mounted only if it does not collide with an already mounted VMFS datastore that has the same UUID (that is, VMFS signature). In other words, if the original LUN that contains the original VMFS datastore together with its snapshot LUN are both presented to the ESXi host simultaneously, force mounting the VMFS datastore from the snapshot LUN on this same ESXi host is not allowed. In this case, the only way to mount that VMFS datastore from the snapshot LUN is to resignature and choose a new label for that datastore.

  • In a datacenter that has multiple ESXi hosts, a VMFS datastore on a snapshot LUN is force mounted on any one of these hosts. When this datastore is mounted on a given host, that datastore cannot be force mounted on any other host using the vCenter Server user interface. To mount that datastore on other hosts, use the preceding steps.

  • If the datastore will not mount correctly in vCenter Server, retry the mount while directly connected to the ESX host. Other hosts see the volume after the resignature operation. If you do not see the volume after the resignature operation, do a manual rescan from each host(s) as required, see Performing a rescan of the storage on an ESXi host (broadcom.com)

 

Identify if a datastore is force mounted on an ESXi host (broadcom.com)

Extending or increasing a datastore through vCenter Server fails (broadcom.com)

vSphere でスナップショット LUN として検出された LUN のトラブルシューティング

 

Powered by Wolken Software