Failed to expand VMFS datastore - Failed to update disk partitions
search cancel

Failed to expand VMFS datastore - Failed to update disk partitions

book

Article ID: 390604

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • To increase datastore capacity from vCenter, the Increase Datastore Capacity wizard do not list any available LUNs.
  • Expanding datastore from host client fails with error:

    Error:  Failed to expand VMFS datastore Datastore_Name - Failed to update disk partitions for /vmfs/devices/disks/naa.###########################.

  • When expanding the datastore from the vCenter or host client, the storage device backing the datastore will reflect the increased LUN size after the LUN size is modified from the storage array.

 

Validation:

  • From the summary page of a datastore on a host client the lun id can be obtained:

  • The number of extents used by the datastore can be obtained through CLI, by running this command: 

# vmkfstools -Ph -v 10 /vmfs/volumes/Datastore_Name

VMFS-5.61 (Raw Kajor Version: 14) file system spanning 3 partitions.
File system label (if any) :  Datastore_Name
Mode: public ATS-only
apacity 24 TB, 9 MB available, file block size 1 MB, max supported file size 62.9 TB
Volume Creation Time: Thu Jan 16 09:48:54 2020
Files [max/free) : 130000/129809
Per Blocks (max/free) : 64512/39881
Bub Blocks (max/free) : 32000/31974
econdary Ptr Blocks [max/free) : 256/256
File Blocks (overcommit/used/overcommit 4) : 0/25165047/0
(overcomit/used/overcommit $) : 0/24631/0
(overcomit/used/overcommit 4) : 0/26/0
Volume Metadata size: 932544512
Diak Block Size: 512/512/0
JUID: ########-########-####-############
Partitions spanned (on "ivm") :
naa. ################################:1
naa. ################################:1
naa. ################################:1
Unable to connect to vaai-nasd socket [No such file or directory]
s Native Snapshot Capable: NO
OBJLIB-LIB: Objlib cleanup done.
ORKER: asyncOps=0 maxActiveOps=O maxPending=O maxCompleted=O

  • This error is returned while checking the usable sectors or partition table with getptbl command on the storage device partition table:

# partedUtil getUsableSectors /vmfs/devices/disks/naa.####################

Error: The backup GPT table is not at the end of the disk, as it should be.This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup) ?
Warning: Not all of the space available to /dev/disks/naa.###################### appears to be used, you can fix the GPT to use all of the space (an extra 8589934592 blocks) or continue with the current setting? This will also move the backup table at the end if it is not at the end already. diskSize (21474836480) AlternateLBA (12884901887) LastUsableLBA (12884901854) NewLastUsableLBA (21474836446)
34 21474836446

 

Environment

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Cause

  • This issue can be caused by the backup GPT table not at the end of the disk after backend storage lun size is increased

  • GPT is a partitioning scheme that allows disks larger than 2 TB and provides redundancy with a primary GPT table (located at the start of the disk) and a backup GPT table (located at the end of the disk).

  • The backup GPT table is supposed to be placed at the very last sector of the disk, ensuring redundancy in case the primary table is damaged. The backup table is meant to be a copy of the main partition table located at the end of the disk.

  • The error indicates backup GPT table is not where it should be, which can indicate corruption, an incorrect partitioning layout, or a misalignment issue on the metadata level or on the partition level of the LUN. 

  • The message about "another operating system believes the disk is smaller" suggests that there may be a misalignment between how the disk is being interpreted by the system. 

  • This usually occurs when a disk has been expanded, and the backup GPT table has not been updated to reflect the new size. When a disk is expanded (e.g., after adding more physical disks to a LUN from storage array side), the system needs to update the backup GPT table to reflect the new size.

  • If the backup GPT table has not been moved to the end of the expanded disk or update the backup GPT table, the system might be confused about the disk's size, leading to this error message.

  • As this happens on metadata of the LUN and ESXi host does not have write access on it, hence vmkernel does not write anything on the log regarding the same. 

  • An inconsistent LUN number could be the potential cause of this issue. Ensure that the LUN number is consistent across all ESXi hosts where the datastore is mounted.

    For example, in the image below, the LUN number differs across the ESXi hosts connected to the datastore.

Note: The LUN ID should be assigned to the LUN from the storage array during mapping to the ESXi hosts, and it must remain consistent for the same LUN across all ESXi hosts.

 

Resolution

We may take the following steps to resolve the issue:

  1. Gather the naa ID of the datastore 
    esxcfg-scsidevs -m | grep DatastoreName
    naa.################################:1 /vmfs/devices/disks/naa.################################:1 ########-e4ad8cd2-6157-############ 0 DatastoreName
     
  2. Check the existing partition table and if necessary run FixGPT, run a VMFS rescan and try to expand the datastore.
    partedUtil getptbl /vmfs/devices/disks/naa.################################

    • If you encounter the following, run FixGPT.

      Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by moving the backup to the end (and removing the old backup)?
      Warning: Not all of the space available to /dev/disks/naa.naa.################################ appears to be used, you can fix the GPT to use all of the space (an extra 390619136 blocks) or continue with the current setting? This will also move the backup table at the end if is is not at the end already. diskSize (1367171072) AlternateLBA (976551935) LastUsableLBA (976551902) NewLastUsableLBA (1367171038)
      gpt
      85102 255 63 1367171072
      1 2048 976549888 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

      partedUtil fixGpt /vmfs/devices/disks/naa.naa.################################
      FixGpt tries to fix any problems detected in GPT table.
      Please ensure that you don't run this on any RDM (Raw Device Mapping) disk.
      Are you sure you want to continue (Y/N): Y
      Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by moving the backup to the end (and removing the old backup)?
      Fix/Ignore/Cancel? Fix
      gpt
      85102 255 63 1367171072
      1 2048 976549888 ##################### vmfs 0

    • Verify that the error has been fixed

      partedUtil getptbl /vmfs/devices/disks/naa.naa.################################
      gpt
      85102 255 63 1367171072
      1 2048 ##################### vmfs 0

    • Run a VMFS rescan

      vmkfstools -V

    • Expand the datastore from vCenter. Refer to Increase Capacity of vSphere VMFS Datastore
    • If the datastore expansion through vCenter fails, expand the datastore from the host client. Refer to Extending or increasing a datastore through vCenter Server fails
    • If both fails, proceed to the next step.
       
  3. Manually extend the size of the datastore

    • Identify the desired ending sector for the VMFS datastores partition

      partedUtil getUsableSectors /vmfs/devices/disks/naa.naa.################################
      34 1367171038

    • Resize the partition containing the target VMFS Datastore. Specify the existing starting sector of the partition and the desired ending sector.

      Note: The New Ending Sector here would be 1 sector less than output of the total usable sector obtained in the last step.  New Ending sector > 1367171038 - 1 = 1367171037

      partedUtil resize "/vmfs/devices/disks/naa.naa.################################" 1 2048 1367171037

    • Grow the VMFS Datastore in to the new space. Specify the partition containing the target VMFS Datastore twice.

      vmkfstools --growfs "/vmfs/devices/disks/naa.naa.################################:1" "/vmfs/devices/disks/naa.naa.################################:1"

    • Run VMFS rescan

      vmkfstools -V

If the symptoms above align with your issue, please contact Broadcom Support by opening a new support ticket for assistance in validating the partition size and grow the filesystem on the newly added capacity.

Refer this link for creating and managing Broadcom support cases: Creating and Managing Broadcom Support cases

Additional Information

To validate the partition table on the storage device. Refer to this article for more information: https://knowledge.broadcom.com/external/article/321398/failed-to-expand-vmfs-datastore-cannot.html

Powered by Wolken Software