AppVols file copies and App delete operations failing to remove VMDKs or copy across datastores

Products

• VMware vCenter Server

Issue/Introduction

Deleting Apps via the AppVols console fails to remove the VMDK file and it cannot be removed through the datastore browser in the vCenter. The hostd.log shows a NO_PERMISSIONS error.

vCenter logs show a failure (vpxd.log):

YYYY-MM-DDTHH:MM:SS.SSS-06:00 warning vpxd[06150] [Originator@6876 sub=Vmomi opID=########-#######-auto-####-##:########-##] VMOMI activation LRO failed; <<########-####-####-####-############, <TCP '127.0.0.1 : 8085'>
, <TCP '127.0.0.1 : 48968'>>, FileManager, vim.FileManager.delete, <vim.version.v8_0_3_0, internal, 8.0.3.0>, (null)>, N3Vim5Fault16CannotDeleteFile9ExceptionE(Fault cause: vim.fault.CannotDeleteFile
--> )
--> [context]----------  REMOVED ------------------------------=[/context]
YYYY-MM-DDTHH:MM:SS.SSS-06:00 info vpxd[06150] [Originator@6876 sub=vpxLro opID=########-#######-auto-####-##:########-##] [VpxLRO] -- FINISH task-#######
YYYY-MM-DDTHH:MM:SS.SSS2-06:00 error vpxd[06150] [Originator@6876 sub=Default opID=########-#######-auto-####-##:########-##] [VpxLRO] -- ERROR task-####### -- ########-####-####-####-############
#-##########) -- FileManager -- vim.FileManager.delete: :vim.fault.CannotDeleteFile
--> Result:
--> (vim.fault.CannotDeleteFile) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    file = "[<FilePath/FileName>.vmdk"
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f5604740b08, h:298, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-1084/vpxa>]: Delete
--> "
--> }
--> Args:
-->
--> Arg datacenter:
--> 'vim.Datacenter:########-####-####-####-############:datacenter-#'
--> Arg datastorePath:
--> "[DATASTORE] FilePath/<FileName>.vmdk"
--> Arg fileType:
--> "VirtualDisk"

hostd.log:
----------
YYYY-MM-DDTHH:MM:SS.SSSZ In(166) Hostd[2099849]: [Originator@6876 sub=Libs opID=UnknownOpId-################-TicketID:########-####-####-####-############ ExpirationDate:############## RemainingUseCount:0] [NFC ERROR]NfcFssrvr_InitConnection: Failed to open connection: The session does not have the required permissions (NFC_NO_PERMISSION)
YYYY-MM-DDTHH:MM:SS.SSSZ In(166) Hostd[2099849]: [Originator@6876 sub=Libs opID=UnknownOpId-################-TicketID:########-####-####-####-############ ExpirationDate:############## RemainingUseCount:0] [NFC ERROR]NfcProcessStreamMsg: Could not process NFC_FSSRVR_OPEN message: NFC_NO_PERMISSION (The session does not have the required permissions)
YYYY-MM-DDTHH:MM:SS.SSSZ Er(163) Hostd[2099849]: [Originator@6876 sub=Nfcsvc opID=UnknownOpId-################-TicketID:########-####-####-####-############ ExpirationDate:############## RemainingUseCount:0] Read error from the nfcLib: NFC_NO_PERMISSION (done = yep)
...
YYYY-MM-DDTHH:MM:SS.SSSZ In(166) Hostd[2099849]: [Originator@6876 sub=Libs opID=UnknownOpId-################-TicketID:########-####-####-####-############ ExpirationDate::############## RemainingUseCount:0] [NFC INFO]NfcSessionStats: session=#########, type=server, clientName='UnknownClient', streamMode=0, fssrvrMode=1, aioMode=0, version=11, remoteVersion=11, currState=NFC_IDLE, prevState=NFC_FSSRVR_OPENING, returnCode=NFC_NO_PERMISSION (11), detail="The session does not have the required permissions -- File open failed: File not open", opID='UnknownOpId', sessionDurationUs=218991, sessionIdleTimeUs=216691 (98.9%), fileIoIdleTimeUs=218970 (100.0%), attemptedFileTransfers=0, successfulFileTransfers=0, totalBytesTransferred=0, totalFilesSize=0, fileLockFailureCount=0

NOTE: The above is true even with the NFS volume mounted correctly with no_root_squash set.

Environment

vSphere with AppVols

Cause

The permissions to access the vmdk descriptor to set the ddb.deletable variable correctly and allow the deletion.

Resolution

Unmount the NFS AppVols datastores from any ESX host that is not running AppVols.