Symptoms:
- Uploads to catalog fail with the error "Cannot complete login due to an incorrect user name or password. - Cannot complete login due to an incorrect user name or password."
- The upload "Imported Media File" task may also hang indefinitely without an error being observed.
- Catalog uploads previously worked successfully without issue.
- Within the /opt/vmware/vcloud-director/logs/vcloud-container-debug.log you observe an error similar to the following:
2022-07-20T11:40:26,879 | ERROR | task-service-activity-pool-156497 | FutureUtil | Error creating Media | requestId=<REQUEST_UUID>,request=POST https://vcloud.example.com/api/catalog/<CATALOG_UUID>/action/upload,requestTime=1595364143157,remoteAddress=<IP>:<PORT>,userAgent=Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:77.0) Gecko/20100...,accept=application/*+xml;version 34.0 vcd=<VCD_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_UUID>)
com.vmware.ssdc.util.LMException: Cannot complete login due to an incorrect user name or password.
at com.vmware.ssdc.util.LMException.wrap(LMException.java:128)
at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:110)
at com.vmware.vcloud.common.activity.AbstractActivity$CancelablePhase.invoke(AbstractActivity.java:365)
at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot complete login due to an incorrect user name or password.
at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$TransferWrapper.execute(NfcClientFacadeImpl.java:176)
at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl.putFile(NfcClientFacadeImpl.java:233)
at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:107)
Caused by: HTTP error on PUT https://vcloud.example.com/folder/<DATASTORE_UUID>/media/<ORGANIZATION_VDC_UUID>/<ORGANIZATION_UUID>/media-<MEDIA_UUID>.iso?dcPath=example-dc&dsName=example-dcc01-vsan: 500
- Within the vCenter Server /var/log/vmware/vxpd/vpxd.log the following error is visible:
2022-07-20T11:40:18.454Z warning vpxd[04567] [Originator@6876 sub=Default] [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 14 from server: There is no disk space available -- NfcFssrvrWriteCB: Failed to write 1024000 bytes @ 38912000 : There is no space left on the device
- Within the ESXi /var/log/vpxa.log the following error is visible:
2022-07-20T11:40:20.017Z warning vpxa[2104498] [Originator@6876 sub=Libs] [NFC ERROR] NfcFileRawPwrite: Failed to write xxxxx bytes at offset xxxxxx from file '/vmfs/volumes/vsan:<VSAN_UUID>/<DATASTORE_UUID>/media/<ORGANIZATION_VDC_UUID>/<ORGANIZATION_UUID>/media-<MEDIA_UUID>.iso': Success (@&!*@*@(msg.fileio.success)Success)
2022-07-20T11:40:20.017Z warning vpxa[2104498] [Originator@6876 sub=Libs] [NFC ERROR] NfcFssrvrWriteCB Received error:14, fileError:10
2022-07-20T11:40:20.018Z warning vpxa[2102551] [Originator@6876 sub=Libs] [NFC ERROR] NfcFssrvrSendResponse: Found an error in the nfc session: There is no disk space available (NFC_NO_DISKSPACE)
2022-07-20T11:40:20.018Z warning vpxa[2102551] [Originator@6876 sub=Libs] [NFC ERROR] Sending Nfc error 14: There is no disk space available -- NfcFssrvrWriteCB: Failed to write xxxxx bytes @ xxxxxx : There is no space left on the device
- Within the ESXi /var/log/vmkernel.log the following error is visible:
2022-07-20T11:40:44.573Z cpu11:2105484)WARNING: Res3: 7542: '<REQUEST_UUID>': [rt 1] No Space - did not find enough resources after second pass! (needed: 1, found: 0)
2022-09-22T11:40:44.573Z cpu11:2105484)Res3: 7543: '<REQUEST_UUID>': [rt 1] Resources t 0, e 0, np 192, wb 0, b 0, RCs u 0, i 0, nf 1305, pe 0, oe 0
2022-07-20T11:40:44.573Z cpu11:2105484)BC: 410: write to media-<MEDIA_UUID>.iso (f530 28 3 xxxxx 537c4b6e xxxxx 504b1096 xxxxx 2a 0 0 0 0 0) 1048576 bytes failed: No space left on device