VCF Operations Deployment Fails: "Upload binary content to VCF Operations fleet management failed" (401 Bad Credentials)

Products

VCF Operations VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

When attempting to deploy VMware Cloud Foundation (VCF) Operations via VCF SDDC Manager, the deployment task fails at the "Upload VCF Operations binary to VCF Operations fleet management" stage. The task shows the following error:

Description	Upload VCF Operations binary to VCF Operations fleet management
Progress Messages	Upload binary content /nfs/vmware/vcf/nfs-mount/bundle/<directory>/Operations-Appliance-9.x.x.x.<build_number>.ova to VCF Operations fleet management failed
Error	Message: Upload binary content /nfs/vmware/vcf/nfs-mount/bundle<directory>/Operations-Appliance-9.x.x.x.<build_number>.ova to VCF Operations fleet management failed Remediation Message: Reference Token: <TOKEN_NUMBER> Cause: Failed to retrieve binary source mapping details from VCF Operations Management at <FQDN> 401 : "{"code":401,"description":"Bad credentials"}"

Error in /var/log/vmware/vcf/domainmanager/domainmanager.log reveals a 401 UNAUTHORIZED error during the API call to VCF Fleet Management::

DEBUG [vcf_dm,<WORKFLOW_ID>] [c.v.e.s.r.c.LoggingHttpRequestInterceptor,dm-exec-#]  Request URI: https://<FM_FQDN>/lcm/lcops/api/v2/settings/product-binaries
Request method: GET
Request body:
Response code: 401 UNAUTHORIZED
Response headers: [Date:"<DATE_TIME>", Content-Type:"application/json", Content-Length:"44", Connection:"keep-alive", X-Content-Type-Options:"nosniff", X-XSS-Protection:"1; mode=block", Cache-Control:"no-cache, no-store, max-age=0, must-revalidate", Pragma:"no-cache", Expires:"0", X-Frame-Options:"DENY", Content-Security-Policy:"script-src 'self'", Strict-Transport-Security:"max-age=31536000; includeSubDomains", Lcm-API-Version:"8.0"]
Response body: {"code":401,"description":"Bad credentials"}
ERROR [vcf_dm,<WORKFLOW_ID>] [c.v.e.s.v.s.UploadBinaryToVcfOpsMgmtAction,dm-exec-#]  Error in uploading binary content /nfs/vmware/vcf/nfs-mount/bundle/<directory_id>Operations-Appliance-9.x.x.x.<build_number>.ova to VCF Operations Management
com.vmware.evo.sddc.common.vrealize.vrlcm.VrlcmException: Failed to retrieve binary source mapping details from VCF Operations Management at <FM_FQDN>

Environment

VCF SDDC 9.x
VCF Operations 9.x
VCF Fleet Management 9.x

Cause

This issue occurs due to an authentication failure between SDDC Manager and the VCF Operations Fleet Management appliance. The credentials stored in the active SDDC Manager deployment workflow are either incorrect or do not meet the strict password complexity requirements for VCF Operations, resulting in the API request being rejected with a 401 Bad Credentials error.

Resolution

To resolve this issue, you must ensure the passwords meet the required complexity, update the credentials for VCF Fleet Management, and manually inject the corrected passwords into the failed SDDC Manager workflow before retrying the task.

Step 1: Validate Password Complexity

Ensure that the passwords you intend to use for VCF Operations and VCF Fleet Management strictly adhere to the following requirements:
1. Length: >= 15 characters.
2. Complexity: Must contain at least one uppercase letter, one lowercase letter, one number, and one special character.
3. Allowed Special Characters: ! @ # $ % ^ & * ( )

Step 2: Update Passwords in Fleet Management

Manually update and verify the root and admin@local passwords directly on the VCF Fleet Management appliance to ensure they match the verified complex passwords from Step 1.

Step 3: Modify the Failed Workflow in SDDC Manager

Follow the instructions in Re-try an existing workflow by modifying the workflow spec file
Locate and update the following specific parameters with your validated passwords:
1. VCF Operations and VCF Fleet Management Passwords: Search for and update the root and admin passwords for VCF Operations and VCF Fleet Management everywhere they appear in the file.
2. Fleet Management API Password: Locate "vcfOpsLcmApiPassword" and set its value to the admin@local password of Fleet Management.
3. Fleet Management SSH Password: Locate "vcfOpsLcmSshPassword" and set its value to the root password for Fleet Management.

Step 4: Resume the Deployment

Follow the instructions in Re-try an existing workflow by modifying the workflow spec file

Additional Information

Reset the root password in VCF Operations Fleet Management Appliance
admin@local account for Fleet Management component appears as disconnected