Receiving error "Invalid configuration for device '0'" intermittently when attempting to attach App Volumes to a VM

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

While some volume attach operations are successful for App Volumes, some attempts fail outright with an error similar to the following:

"Invalid configuration for device '0'"

Reviewing the ESXi hostd.log reveals the following errors related to the volume in question when attempting to attach to the VM:

YYYY-MM-DD HH:MM:SS info hostd[2116951] [Originator@6876 sub=Libs] UnresolvedVmfsVolume: deviceName=naa.<ID>:1,lvmName=6036cadf-########-####-########5d2e,label=Array_Volume_02(VMFS),fsUuid=6036cadf-########-####-########5d2e
YYYY-MM-DD HH:MM:SS info hostd[2116951] [Originator@6876 sub=Libs] Refresh: Refreshed unmounted volume /vmfs/volumes/6036cadf-########-####-########5d2e. Reseting console path
YYYY-MM-DD HH:MM:SS info hostd[2116951] [Originator@6876 sub=Libs] GetUnmountedVmfsFileSystemsInt: uuid:6036cadf-########-####-########5d2e, Label:Array_Volume_02(VMFS),logicalDevice:6036cadf-########-####-########5d2e,headExtent:naa.514f0c5a16400002:1
YYYY-MM-DD HH:MM:SS info hostd[2103307] [Originator@6876 sub=Libs] OBJLIB-LIB:  Failed to get VCFS root path for '/vmfs/volumes/6036cadf-########-####-########5d2e': No such file or directory (131076).
YYYY-MM-DD HH:MM:SS info hostd[2103307] [Originator@6876 sub=Libs] FILE: FileVMKGetMaxFileSize: Could not get max file size for path: /vmfs/volumes/6036cadf-########-####-########5d2e, error: Inappropriate ioctl for device
YYYY-MM-DD HH:MM:SS info hostd[2103307] [Originator@6876 sub=Libs] OBJLIB-LIB: ObjLib_GetMaxSizeInfo: failed. Obj backend type: file, Path: /vmfs/volumes/6036cadf-########-####-########5d2e, Error: Unknown object error
YYYY-MM-DD HH:MM:SS error hostd[2103307] [Originator@6876 sub=Hostsvc.Datastore] Cannot retrieve max file size from objLib for /vmfs/volumes/6036cadf-########-####-########5d2e, error 25, revert to default
 

Environment

VMware vSphere ESXi 7.x

Cause

In this particular scenario, it was learned that a subset of ESXi hosts in the cluster did not have the appropriate VMFS Volumes mounted:

$ localcli storage filesystem list -i
Mount Point                                        Volume Name             UUID                                 Mounted  Type    Size           Free
----------------------------------------------------------------------------------------------------------------------------------------------------
/vmfs/volumes/6036ce17-########-###b-#########2e  Array_Volume_25         6036ce17-########-###b-#########2e  true     VMFS-6  5497289703424  4600743591936
/vmfs/volumes/6036ce32-########-###c-#########2e  Array_Volume_26         6036ce32-########-###c-#########2e  true     VMFS-6  5497289703424  4603104985088
/vmfs/volumes/6036ce52-########-###5-#########2e  Array_Volume_27         6036ce52-########-###5-#########2e  true     VMFS-6  5497289703424  4607515295744
/vmfs/volumes/6036ce71-########-###1-#########2e  Array_Volume_28         6036ce71-########-###1-#########2e  true     VMFS-6  5497289703424  4635040415744
                                                   Array_Volume_02         6036cadf-########-###4-#########2e  false    VMFS-6              0              0
                                                   Array_Volume_01         6036ca2b-########-###e-#########2e  false    VMFS-6              0              0
/vmfs/volumes/84ada772-########-###6-#########17                          84ada772-########-###6-#########17  true     vfat        261853184       42352640
/vmfs/volumes/cdf9b778-########-###2-##########d7                          cdf9b778-########-###2-##########d7  true     vfat        261853184       42573824
/vmfs/volumes/602ba1cd-########-###5-##########28                          602ba1cd-########-###5-##########28  true     vfat       4293591040     4234018816
/vmfs/volumes/602ba1c4-########-###9-##########28                          602ba1c4-########-###9-##########28  true     vfat        299712512       92250112

From /etc/vmware/esx.conf we can confirm that the volumes are set to "unmounted":

/fs/vmfs[6036ca2b-########-###b-#########2e]/unmounted = "true"
/fs/vmfs[6036cadf-########-###4-#########2e]/unmounted = "true"

Resolution

In this scenario, a user in the past had forcibly unmounted the volume in question but never remounted it. As the unmounting operation saves this action as part of the ESXi host configuration, the volume does not automatically remount on its own after a reboot of the host. A user must manually remount the volume on all applicable ESXi hosts. To do this, the process is outlined in our product documentation:

 

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-storage-7-0.html