Automating OVF deployment from content library fails at random intervals with the error: "Transfer failed: IO error during transfer. Unable to access file since it is locked"
Article ID: 400333
Updated On:
Products
VMware vCenter Server
VMware vSphere ESXi
Issue/Introduction
Automating OVF deployment from content library at scale might fail at random intervals with the error, "unable to access file as it is locked", for the below configuration:
- OVF is configured with multiple disks
- All disks are deployed to the same storage destination. Specifically to a shared VMFS datastore or VMFS datastore cluster.
- API used is, Deploy Ovf Library Item Id
- Sample API with deployment spec:
JSON_PAYLOAD=$(cat << EOF { "deployment_spec": { "accept_all_EULA": false, "name": "$VM_NAME", "default_datastore_id": "$DATASTORE_CLUSTER_ID" }, "target": { "folder_id": "$FOLDER_ID", "resource_pool_id": "$RESOURCE_POOL_ID" } } RESPONSE=$(curl -X POST -k "https://$VCENTER_URL/api/vcenter/ovf/library-item/$OVA_ID?action=deploy" \ -H "vmware-api-session-id: $SESSION_TOKEN" \ -H "Content-Type: application/json" \ -d "$JSON_PAYLOAD")
Note: Deployment API is synchronous and blocking, unless script waits on a thread and facilitate deployments in parallel
The following symptoms are observed:
- vCenter UI : Deploy OVF Template task fail with error description as below (refer attached UI snapshot)
Failed to deploy OVF package. Cause: A general system error occurred: Transfer failed: IO error during transfer of ds:///vmfs/volumes/682d9b76-####-####-005##6af826f//VM-19/VM-19.vmdk: Unable to transfer file /vmfs/volumes/6655aa42-#####-#####-005056##6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk (Unable to access file /vmfs/volumes/6655aa42-#####-#####-005056##6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk since it is locked).
- API : Deploy OVF API return RESPONSE with "succeeded":false and error as below
{"error":{"warnings":[],"information":[],"errors":[{"category":"SERVER","error":{"error_type":"SERVICE_UNAVAILABLE","messages":[{"args":["IO error during transfer of ds:///vmfs/volumes/67976d50-####-####-6c92cf8dfce0//deploy-vm-112/deploy-vm-112.vmdk: Unable to transfer file /vmfs/volumes/67976dbe-####-####-6c92cf8dfce0//contentlib-68e57ce4-####-####-9b71-12f8##75c725/95460d45-064a-####-####-1d576##b5029/###-####-####-r8-20250312-191440-1_23c4f606-0ef6-####-####-74841bf##ec3.vmdk (Unable to access file /vmfs/volumes/67976dbe-####-####-6c92cf8dfce0//contentlib-68e57ce4-####-####-9b71-12f8##75c725/95460d45-064a-####-####-1d576##b5029/###-####-####-r8-20250312-191440-1_23c4f606-0ef6-####-####-74841bf##ec3.vmdk since it is locked)"],"default_message":"Transfer failed: IO error during transfer of ds:///vmfs/volumes/67976d50-####-####-6c92cf8dfce0//deploy-vm-112/deploy-vm-112.vmdk: Unable to transfer file /vmfs/volumes/67976dbe-####-####-6c92cf8dfce0//contentlib-68e57ce4-####-####-9b71-12f8##75c725/95460d45-064a-####-####-1d576##b5029/###-####-####-r8-20250312-191440-1_23c4f606-0ef6-####-####-74841bf##ec3.vmdk (Unable to access file /vmfs/volumes/67976dbe-####-####-6c92cf8dfce0//contentlib-68e57ce4-####-####-9b71-12f8##75c725/95460d45-064a-####-####-1d576##b5029/###-####-####-r8-20250312-191440-1_23c4f606-0ef6-####-####-74841bf##ec3.vmdk since it is locked).","id":"com.vmware.ovfs.ovfs-main.ovfs.transfer_failed"}]}}]},"succeeded":false} - On vCenter server:
/var/log/vmware/content-library/cls.logshow below error symptoms:
202#-0#-##T15:02:13.636Z | DEBUG | 6c0f7501-b0e6-####-####-432ef0##3266-cb-e3 | cls-simple-activity-6 | ImportSessionActivity | Completing import lease ManagedObjectReference: type = HttpNfcLease, value = session[52###d49-e045-eb62-a2c6-a10###e260b9]528a4733-####-####-c389-12fb##d9ff4f, serverGuid = f77##0f4-e692-####-####-6944##1826cd (exception: (vim.fault.OvfImportFailed) { faultCause = (vmodl.fault.SystemError) { faultCause = null, faultMessage = (vmodl.LocalizableMessage) [ (vmodl.LocalizableMessage) { dynamicType = null, dynamicProperty = null, key = com.vmware.ovfs.ovfs-main.ovfs.transfer_failed, arg = (vmodl.KeyAnyValue) [ (vmodl.KeyAnyValue) { dynamicType = null, dynamicProperty = null, key = 0, value = IO error during transfer of ds:///vmfs/volumes/682d9b76-####-####-005##6af826f//VM-19/VM-19.vmdk: Unable to transfer file /vmfs/volumes/6655aa42-#####-#####-005056##6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk (Unable to access file /vmfs/volumes/6655aa42-3df987 2b-a96e-005056af6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk since it is locked) } ], message = Transfer failed: IO error during transfer of ds:///vmfs/volumes/682d9b76-####-####-005##6af826f//VM-19/VM-19.vmdk: Unable to transfer file /vmfs/volumes/6655aa42-#####-#####-005056##6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk (Unable to access file /vmfs/volumes/66 55aa42-3df9872b-a96e-005056af6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk since it is locked). } ], reason = }, faultMessage = null }) - On ESXI host:
/var/run/log/hostd.log- show below error symptoms on NFC copy
202#-0#-##T15:02:06.948Z Er(163) Hostd[525491]: [Originator@6876 sub=NfcManager opID=882###53-5365-####-####-3f9c18###05e-50-0-c8cb sid=526##8c3 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-776d4f08-eac3-4689-a5cc-6c19fa206e24] File transfer [/vmfs/volumes/6655aa42-#####-#####-005056##6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk -> /vmfs/volumes/682d9b76-####-####-005##6af826f//VM-19/VM-19.vmdk] failed: The file is locked or in use -- NfcFile_Clone: Failed. 202#-0#-##T15:02:06.949Z Er(163) Hostd[525491]: [Originator@6876 sub=NfcManager opID=882###53-5365-####-####-3f9c18###05e-50-0-c8cb sid=526##8c3 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-77##4f08-####-####-a5cc-6c19f####e24] Error encountered while processing copy spec for file [/vmfs/volumes/6655aa42-#####-#####-005056##6332//contentlib-bca044dd-####-####-####-daa95d##fc52/695341a1-####-#####-#####-8fa21##35b1f/Template_01-1_37c85a50-####-####9c3b-23#####657a.vmdk -> /vmfs/volumes/682d9b76-####-####-005##6af826f//VM-19/VM-19.vmdk]: 202#-0#-##T15:02:06.950Z Er(163) Hostd[525467]: --> N3Vim5Fault10FileLocked9ExceptionE(Fault cause: vim.fault.FileLocked 202#-0#-##T15:02:06.950Z Er(163) Hostd[525467]: --> ) 202#-0#-##T15:02:06.950Z Er(163) Hostd[525467]: --> [context]zKq7AVICAgAAAO6NbgEQaG9zdGQAAOPBR2xpYnZtYWNvcmUuc28AASR5XWhvc3RkAAH52VEByMubATKknAHoB50BbA2dAR/1mwEr8JoCnckCbGlibmZjLXR5cGVzLnNvAAG1lWIAHtMsAOD3LAA7SFIDUngAbGlicHRocmVhZC5zby4wAAQPMg9saWJjLnNvLjYA[/context] 202#-0#-##T15:02:06.950Z In(166) Hostd[525491]: [Originator@6876 sub=Libs opID=882###53-5365-####-####-3f9c18###05e-50-0-c8cb sid=526##8c3 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-77##4f08-####-####-a5cc-6c19f####e24] [NFC INFO]Nfc_CloseSessionEx: session=557###790 202#-0#-##T15:02:06.950Z In(166) Hostd[525491]: [Originator@6876 sub=Libs opID=882###53-5365-####-####-3f9c18###05e-50-0-c8cb sid=526##8c3 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-77##4f08-####-####-a5cc-6c19f####e24] [NFC INFO]NfcUtil_DestroyTimer: Removed periodic timer(5573A98F20) for session=557###790, OpID: 882###53-5365-####-####-3f9c18###05e-50-0-c8cb. 202#-0#-##T15:02:06.950Z In(166) Hostd[525491]: [Originator@6876 sub=Libs opID=882###53-5365-####-####-3f9c18###05e-50-0-c8cb sid=526##8c3 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-77##4f08-####-####-a5cc-6c19f####e24] [NFC INFO]NfcSessionStats: session=557###790, type=local, clientName='VMware ESX, NfcManager.copy', streamMode=1, fssrvrMode=0, aioMode=0, version=11, remoteVersion=0, currState=NFC_IDLE, prevState=NFC_IDLE, returnCode=NFC_FILE_LOCKED (13), detail="The file is locked or in use -- NfcFile_Clone: Failed.", opID='882###53-5365-####-####-3f9c18###05e-50-0-c8cb', sessionDurationUs=27558174, sessionIdleTimeUs=27558148 (100.0%), fileIoIdleTimeUs=27558148 (100.0%), attemptedFileTransfers=0, successfulFileTransfers=0, totalBytesTransferred=0, totalFilesSize=0, fileLockFailureCount=0 202#-0#-##T15:02:06.950Z Er(163) Hostd[525491]: [Originator@6876 sub=NfcManager opID=882###53-5365-####-####-3f9c18###05e-50-0-c8cb sid=526##8c3 user=vpxuser:VSPHERE.LOCAL\vpxd-extension-77##4f08-####-####-a5cc-6c19f####e24] Copy operation failed with error: N3Vim5Fault10FileLocked9ExceptionE(Fault cause: vim.fault.FileLocked On ESXI host: /var/run/log/vmkernel.log
202#-0#-##T15:01:59.999Z In(182) vmkernel: cpu1:525508 opID=79dde6f8)Fil3: 5174: Lock failed on file: VM-19 on vol 'iscsi#4' with FD: <FD c## r49> 202#-0#-##T15:02:13.816Z In(182) vmkernel: cpu2:525492 opID=b45cc020)VmMemXfer: vm 525492: 2448: Evicting VM with path:/vmfs/volumes/682d9b76-####-####-005##6af826f/VM-19/VM-19.vmx
Environment
vCenter Server 7.x
vCenter server 8.x
ESXi 7.x
ESXi 8.x
Resolution
Broadcom is aware of current product design limitation in this specific scenario.
This limitation will be reviewed further, and Broadcom will conduct a thorough analysis to determine the best course of action for the future.
To work around the issue, when using the Deploy Ovf Library Item Id API,
- If the API response reports the error, "Unable to access file /#/#/#/#.vmdk since it is locked", wait for the failed VM to be automatically cleaned up.
- After the failed VM is cleaned up, retry the OVF deployment once again.
NOTE: Failed VM will be automatically deleted and storage cleaned up as part of API handling, there are no additional steps required to clean up the failed VM.
Feedback
Yes
No
Powered by
