ESXi host upgrade fails with the error message "Failed to query file system stats: Errors:"

Products

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• ESXi host upgrade fails with the error message "Failed to query file system stats: Errors:"
• During ESXi command line upgrade fails with:

  [InstallationError]
   Failed to query file system stats: Errors:
   Error getting data for filesystem on '/vmfs/volumes/###########-###########-####-##################': Cannot open volume: /vmfs/volumes/###########-###########-####-##################, skipping.
        cause = Errors:
   Error getting data for filesystem on '/vmfs/volumes/###########-###########-####-##################': Cannot open volume: /vmfs/volumes/###########-###########-####-##################, skipping.
   Please refer to the log file for more details.

• VIB upgrade/update via CLI fails with the following error message:

  [InstallationError]
   Failed to query file system stats: Errors:
   Cannot open volume: 
        cause = Errors:
  Cannot open volume:
   Please refer to the log file for more details.

• The following snippets will be observed in the /var/run/log/esxupdate.log:

  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR: Traceback (most recent call last):
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Utils/HostInfo.py", line 225, in _getFileSystemList
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:     return runCli(cmd, True)
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:   File "/lib64/python3.8/site-packages/esxutils.py", line 220, in runCli
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR: esxutils.EsxcliError: Errors:
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR: Cannot open volume:
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR: During handling of the above exception, another exception occurred:
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR: Traceback (most recent call last):
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 772, in <module>
  YYYY-MM-DDTHH:MM:SSZ esxupdate: #######: root: ERROR:     main()

• Running the command "df -h" on the affected ESXi host will result in the same error message: "Cannot open volume"

Environment

VMware vSphere ESXi 8.0.x

Cause

This issue occurs when an ESXi host is attempting to access a datastore that is no longer correctly presented or accessible, often caused by an incorrectly unmounted or "stale" datastore reference.

Resolution

Follow these steps to identify and resolve the storage presentation issue:

1. Verify Datastore Accessibility:
   1. Log in to the ESXi host via SSH.
   2. Run the following command to check for inaccessible volumes: esxcli storage filesystem list
   3. Confirm if the datastore shows as inaccessible or if the command itself returns the "Cannot open volume" error.
2. Rescan Storage: Ref :Performing a rescan of the storage on an ESXi host
   1. Perform a rescan of the storage adapters to refresh the host's view of available datastores: esxcli storage core adapter rescan -a
   2. Rescan filesystems: vmkfstools -V
3. Clear Hostd Cache:
   • Restart the host management services to clear potential cached metadata errors: /etc/init.d/hostd restart
4. Reattempt Upgrade:
   1. After the rescan and service restart, verify that the datastore is present in the list of datastores for the host.
   2. If the datastore is presented properly, reattempt the upgrade.
   3. If the failure persists, reboot the host to ensure all stale references are cleared, then retry the upgrade.

Additional Information

NSX Fails to Install on ESXi Host "Failed to query file system stats: Errors:"