VCF Automation deployment fails with error "Failed to obtain SDDC Manager token"

Products

• VCF Automation

Issue/Introduction

• When deploying VCF Automation, the process fails at the "Start component installation using SDDC lifecycle service" step with the following error:
  
  An unexpected error occurred in step deploy_vsp_cluster_task_ref. Reference Code: <reference code id>. Please contact support with this reference code. Detail: Failed to obtain SDDC Manager token for: 'https://<SDDC Manager FQDN>:443'.
  
  
• In the fleet-lcm-service pod logs, the following entries are observed:
  
  Could not get token for user account svc-vcf-sddc-1cm-SDDC_MANAGER-xxx
Failed to register FDS as service config com.broadcom.vcf.sddcmanager.client.exception.AcquireSddcManagerTokenException: Failed to obtain SDDC Manager token for: 'https://<SDDC Manager>: 443'.
  
  
• After multiple failed retry attempts, the client IP address becomes blocked, giving below errors in logs:
  
  Body: {"errorCode":"CLIENT IP BLOCKED","arguments":[], "message":"Client IP is blocked.", "referenceToken": " *** MASKED *** "}

Environment

VMware Cloud Foundation 9.1
VCF Automation 9.1

Cause

If SDDC Manager service account password rotation is executed two or more times (whether via scheduled or on-demand rotation), the service account credentials within SDDC Manager can be inadvertently deleted. This causes the credentials stored in SDDC Lifecycle Management (LCM) to become stale and unauthenticated.

Resolution

A new dedicated service account must be generated via a script to replace the stale service account credentials in SDDC LCM.

Please open a support request with Broadcom Support and reference this knowledge base article. A technical support representative will assist in executing the required script to provision the replacement service account and resume deployment.