Error: "Cannot complete login due to an incorrect user name or password" when downloading an ISO file from a VMware Cloud Director catalog

Products

• VMware Cloud Director

Issue/Introduction

• After clicking the download button on an ISO hosted in a VMware Cloud Director (VCD) catalog, the "Enabled Download of media file" task fails with the error

Internal Server Error - Cannot complete login due to an incorrect user name or password

• Within the log file /opt/vmware/vcloud-director/logs/vcloud-container-debug.log the following error is observed:

2022-08-11 11:13:19,779 | DEBUG    | storage-fabric-activity-pool-12 | NfcClientFacadeImpl            | Unauthorized session still occurred after retrying with a new session on https://vcenter.example.com/sdk, abandon operation. | requestId=<Request ID>,request=POST https://vcloud.example.com/api/media/<Media ID>/action/enableDownload,requestTime=1660216392659,remoteAddress=#.#.#.#:56536,userAgent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...,accept=application/*+xml;version 36.2 vcd=<VCD ID>,task=<Task ID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<UUID>) vcd=<VCD ID>,task=<Task ID> activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:<UUID>) activity=(com.vmware.vcloud.fabric.storage.media.impl.ExportMediaActivity,urn:uuid:<UUID>) activity=(com.vmware.vcloud.fabric.storage.copy.impl.DownloadDatastoreFileActivity,urn:uuid:<UUID>)
HTTP error on GET https://vcenter.example.com/folder/<VCD Folder Name>/media/<Media Path ID>/media-<Media ID>.iso?dcPath=<Datacenter Name>&dsName=<Datastore Name>: 500
        at com.vmware.vcloud.val.nfc.client.impl.HttpTransferClientSession$StreamConsumer.execute(HttpTransferClientSession.java:116)
        at com.vmware.vcloud.val.nfc.client.impl.HttpTransferClientSession.getFile(HttpTransferClientSession.java:154)

2022-08-11 11:13:19,806 | DEBUG    | storage-fabric-activity-pool-12 | DownloadDatastoreFileActivity  | [Activity Execution] Serializing Future: /<VCD Folder Name>/media/<Media Path ID>/media-<Media ID>.iso-> java.util.concurrent.FutureTask@25b25d29[Completed exceptionally: com.vmware.ssdc.util.LMException: Cannot complete login due to an incorrect user name or password.] - Handle: urn:uuid:<UUID>, Current Phase: DownloadDatastoreFileActivity$FileCopyPhase | requestId=<Request ID>,request=POST https://vcloud.example.com/api/media/<Media ID>/action/enableDownload

• Within the ESXi log file /var/log/vmware/hostd.log the following error is observed:
  
  [NFC ERROR]Sending Nfc error 13: NfcFssrvrOpen: Failed to open '[<Datastore Name>]<VCD Folder Name>/media/<Media Path ID>/media-<Media ID>.iso': The file is locked or in use (NFC_FILE_LOCKED)

Environment

VMware Cloud Director 10.x

Cause

This error occurs in Cloud Director when downloading an ISO file which is mounted to a powered on VM. The HTTP GET issued by Cloud Director is unable to retrieve the ISO stored on the VMFS datastore.
This is a result of the new restrictions introduced since vSphere 7.0.x and later releases of vSphere 6.7.x such as 6.7 Update3o. The changes made impact on Cloud Director's ability to carry out cp and HTTP GET commands against files mounted to powered on VMs.

 Should you have additional questions, please engage the vSphere Support Team by creating a ticket with Broadcom Support.

Resolution

This is the expected behavior in Cloud Director with vSphere 7.0 and later versions.
To resolve the issue, one of these actions can be carried out in Cloud Director to enable download of the file:

1. Unmount the ISO file from any VM it's mounted to as outlined here.
2. Power off any VM the ISO is mounted to as outlined here.