app volumes will not mount on VMs - Duplicate directory appvolumes on vSAN datastore

Products

• VMware vSAN

Issue/Introduction

Appvolumes are not mounting on VMs. 

Looking in the vSAN datastore files though vCenter datastore browser you may see two folders with the name "appvolumes" but they point to different UUIDs 

vCenter error may show "invalid configuration on device"

From hostd you see messages like the following:

 error hostd[2103496] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:#######################/#############/ - #####.vmx opID=29b446dd-01-fd-d1fe user=vpxuser:VSPHERE.LOCAL\<USER>] ValidateKeyTypeInSpec: Failed to open '/<PATH TO VMDK>': The system cannot find the file specified (25).
 info hostd[2103496] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/vsan:#######################/#############/ - ####.vmx opID=29b446dd-01-fd-d1fe user=vpxuser:VSPHERE.LOCAL\<USER>] Reconfigure failed: N3Vim5Fault17InvalidDeviceSpec9ExceptionE(Fault cause: vim.fault.InvalidDeviceSpec
--> )

Environment

VMware App Volumes

vSAN

Resolution

If we are unsure what UUID name space has the correct data and want to avoid deleting data before we confirm the issue is full fixed use Method 1 or open a support case with Broadcom and Omnissa for joint collaboration 

Method 1:
Adjust the attributes on both objects and point them to a new path and name

 /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID1> -d appvolumes1 -n appvolumes1
 /usr/lib/vmware/osfs/bin/objtool setAttr -u <Object UUID2> -d appvolumes2 -n appvolumes2

Confirm there is no longer a folder called appvolumes through vCenter datastore browser

Create a new folder using vCenter datastore browser called appvolumes and copy all content for the above two namesapce UUIDs to this new folder using vCenter datastore browser. 
You may need to use the App Volumes console to preform an import but it is not always required 
Review Omnissa documents And reach out to Omnissa support for more assistance with App Volumes console 

Confirm issue is resolved and all content is available then remove the namespace UUID volumes for appvolumes1 and appvolumes2 
This can be done through the vCenter datastore browser

Method 2:

Confirm what namespace UUID holds the correct data (copy any new data over to this namespace UUID if needed) and then delete the un-needed UUID name space through vCenter datastore browser or host UI datastore browser. 
you may need to use the App Volumes console to preform an import but it is not always required 

Review Omnissa documents And reach out to Omnissa support for more assistance with App Volumes console