Partition table fails to update active partitions- VMware vSphere ESXi.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Partition tables on an ESXi/ESX host fail to update
  
  
• Datastores are missing from the VI/vSphere Client.
  
  
• Datastores are missing from /vmfs/volumes/
  
  
• In the /var/log/vmkernel log file, you see entries similar to:
  

  Partition vmhbax:x:x is active: partition table was not updated
YYYY-MM-DDTHH:MM:SS esx1 vmkernel: YYYY-MM-DDTHH:MM:SS cpu9:4119)WARNING: Partition: 806: Partition 1 is active, failure to update partition table for naa.60a9XXXX057396XXXXX6f4b356eXXXXX

• In the /var/log/vmware/hostd.log log file, you see entries similar to:
  

  [YYYY-MM-DDTHH:MM:SS F59A1B90 verbose 'FSVolumeProvider'] RefreshVMFSVolumes called
[YYYY-MM-DDTHH:MM:SS F59A1B90 verbose 'FSVolumeProvider'] RescanVmfs called
[YYYY-MM-DDTHH:MM:SS F59A1B90 verbose 'FSVolumeProvider'] LookupVmfs: Cannot find VMFS volume with id <unset>, path, /vmfs/volumes/4926c7f4-########-####-##########0b, or name <unset>
[YYYY-MM-DDTHH:MM:SS F59A1B90 verbose 'Hostsvc::DatastoreSystem'] ExpungeMissingDatastore: Removing datastore DSNAME.
[YYYY-MM-DDTHH:MM:SS F59A1B90 verbose 'DatastoreBrowser'] 4926c7f4-########-####-##########0b-datastorebrowser::Destroy
[YYYY-MM-DDTHH:MM:SS F59A1B90 verbose 'Hostsvc::Datastore'] DatastoreImpl 4926c7f4-########-####-##########0b Stopped

Note: The timestamps of these error messages represent the time at which log rescans discovered the missing partition tables for the VMFS datastores and not the time when the partitions themselves were removed or cleared. Since the exact time of the partitions being removed or cleared is not shown in the logs, understanding what previous changes were made in the environment can help to identify a root cause.

Environment

 

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Resolution

These errors occur when partition tables for VMFS volumes are removed while ESX/ESXi is using them. This can result from other systems clearing partition tables from visible LUNs to prepare them for use.

For example:

• During installation of a new operating system on a physical server, it may delete or overwrite partitions on those LUNs as part of the installation.
• After an operating system is installed, it can be directed to overwrite partitions of LUNs it has visibility to.
• The ESX installer can be directed to overwrite an existing VMFS Datastore, though it will warn against possible data loss.
• Unattended or automated installations, such as kickstart, jumpstart, and FAI, or full-disk imaging processes may overwrite contents of LUNs that they have access to.
• Utilities such as fdisk, parted or dd may be invoked on any server with LUN visibility, and remove or alter the partition table on the LUN.
• Storage array issues may impact contents of LUNs.

These or similar situations may accidentally occur if LUNs are presented to a mixture of ESX/ESXi and non-ESX/ESXi servers. To prevent against the above situations, restrict LUN visibility to only servers which are trusted and which require access to the VMFS Datastores.

To recreate a partition table, see:

• Using the partedUtil command line utility on ESXi/ESX

Warning: If changes beyond the compromised partition table are made to a disk, data may not be recoverable and need to be restored from backup.