Error: "Cannot complete login due to an incorrect user name or password" when uploading a media file to a vSAN backed catalog in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• Uploading media files to a catalog fail with the error:
  
  Cannot complete login due to an incorrect user name or password. - Cannot complete login due to an incorrect user name or password.

  
• The upload Imported Media File task may also hang indefinitely without an error being observed.
• Catalog uploads previously worked successfully without issue.
• In /opt/vmware/vcloud-director/logs/vcloud-container-debug.log on VMware Cloud Director (VCD), you see an error similar to:
  
  YYYY-MM-DDTHH:MM:SS,879 | ERROR    | task-service-activity-pool-156497 | FutureUtil | Error creating Media | requestId=<REQUEST_UUID>,request=POST https://vcloud.example.com/api/catalog/<CATALOG_UUID>/action/upload,requestTime=1595364143157,remoteAddress=<IP>:<PORT>,userAgent=Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:77.0) Gecko/20100...,accept=application/*+xml;version 34.0 vcd=<VCD_UUID>,task=<TASK_UUID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<TASK_ACTIVITY_UUID>)
com.vmware.ssdc.util.LMException: Cannot complete login due to an incorrect user name or password.
        at com.vmware.ssdc.util.LMException.wrap(LMException.java:128)
        at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:110)
        at com.vmware.vcloud.common.activity.AbstractActivity$CancelablePhase.invoke(AbstractActivity.java:365)
        at com.vmware.vcloud.activity.executors.ActivityRunner.runPhase(ActivityRunner.java:175)
        at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:112)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot complete login due to an incorrect user name or password.
        at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl$TransferWrapper.execute(NfcClientFacadeImpl.java:176)
        at com.vmware.vcloud.val.nfc.client.impl.NfcClientFacadeImpl.putFile(NfcClientFacadeImpl.java:233)
        at com.vmware.vcloud.val.filetransfer.impl.FileTransferToRemoteHostActivity$TransferToRemoteHostPhase.invokeCancelableOperation(FileTransferToRemoteHostActivity.java:107)
Caused by: HTTP error on PUT https://vcloud.example.com/folder/<DATASTORE_UUID>/media/<ORGANIZATION_VDC_UUID>/<ORGANIZATION_UUID>/media-<MEDIA_UUID>.iso?dcPath=example-dc&dsName=example-dcc01-vsan: 500
  
  
• In /var/log/vmware/vxpd/vpxd.log on the vCenter Server, you see a message similar to:
  
  YYYY-MM-DDTHH:MM:SS.454Z warning vpxd[04567] [Originator@6876 sub=Default] [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 14 from server: There is no disk space available -- NfcFssrvrWriteCB: Failed to write 1024000 bytes @ 38912000 : There is no space left on the device
  
  
• In /var/log/vpxa.log on the ESXi host, you see messages similar to:
  
  YYYY-MM-DDTHH:MM:SS.017Z warning vpxa[2104498] [Originator@6876 sub=Libs] [NFC ERROR] NfcFileRawPwrite: Failed to write xxxxx bytes at offset xxxxxx from file '/vmfs/volumes/vsan:<VSAN_UUID>/<DATASTORE_UUID>/media/<ORGANIZATION_VDC_UUID>/<ORGANIZATION_UUID>/media-<MEDIA_UUID>.iso': Success (@&!*@*@(msg.fileio.success)Success)
YYYY-MM-DDTHH:MM:SS.017Z warning vpxa[2104498] [Originator@6876 sub=Libs] [NFC ERROR] NfcFssrvrWriteCB Received error:14, fileError:10
YYYY-MM-DDTHH:MM:SS.018Z warning vpxa[2102551] [Originator@6876 sub=Libs] [NFC ERROR] NfcFssrvrSendResponse: Found an error in the nfc session: There is no disk space available (NFC_NO_DISKSPACE)
YYYY-MM-DDTHH:MM:SS.018Z warning vpxa[2102551] [Originator@6876 sub=Libs] [NFC ERROR] Sending Nfc error 14: There is no disk space available -- NfcFssrvrWriteCB: Failed to write xxxxx bytes @ xxxxxx : There is no space left on the device
  
  
• In /var/log/vmkernel.log on the ESXi host, you see messages similar to:
  
  YYYY-MM-DDTHH:MM:SS.573Z cpu11:2105484)WARNING: Res3: 7542: '<REQUEST_UUID>': [rt 1] No Space - did not find enough resources after second pass! (needed: 1, found: 0)
YYYY-MM-DDTHH:MM:SS.573Z cpu11:2105484)Res3: 7543: '<REQUEST_UUID>': [rt 1] Resources t 0, e 0, np 192, wb 0, b 0, RCs u 0, i 0, nf 1305, pe 0, oe 0
YYYY-MM-DDTHH:MM:SS.573Z cpu11:2105484)BC: 410: write to media-<MEDIA_UUID>.iso (f530 28 3 xxxxx 537c4b6e xxxxx 504b1096 xxxxx 2a 0 0 0 0 0) 1048576 bytes failed: No space left on device

Environment

VMware Cloud Director 10.x

Cause

This issue occurs when the vSAN namespace object is completely full. 

All catalog media images (ISO files) are uploaded by VMware Cloud Director as file objects into a directory structure under a single folder, which results in the namespace filling up. A namespace object has a maximum size of 255 GB.

Resolution

VMware does not recommend using vSAN storage for catalogs in VMware Cloud Director. Other storage solutions should be utilized for catalogs.

 

Note 1: vSAN 8U1 did introduce a feature which allows expansion of the namespace limitation beyond 255gb. Reference Using PowerCLI to expand vSAN namespace for files larger than 255 gb for further details.

Note 2: vSAN File Service (vSAN FS) is another alternative option. In that use-case vSAN FS would be creating a vSAN FS NFS export. This export would then be mounted to the ESXi hosts as a datastore. Once the datastore was tagged to a storage policy, it could then in turn be used for a Cloud Director catalog.