Failed to authenticate in VC during cluster creation in SDDC manager due to duplicate service accounts

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

 

• Attempt to create a new cluster fails with error:
  
  

  Message: Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server vcenter.sub1.domain.com

  Remediation Message: Make sure that vCenter Server vcenter.domain.com contains the relevant depot base image for the vSphere Lifecycle Manager Cluster Image 

  Reference Token: XXXXX 

  Cause: I/O error on POST request for "https://vcenter.sub1.domain.com/rest/com/vmware/cis/session": {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","messages":[{"args":[],"default_message":"Authentication required.","id":"com.vmware.vapi.endpoint.method.authentication.required"}],"challenge":"Basic realm=\"VAPI endpoint\",SIGN realm=94d0944913f0376e021aae6831a1807f5661b25f03aa889f3e709f1ef92f5b4b,service=\"VAPI endpoint\",sts=\"https://vcenter.sub1.domain.com/sts/STSService/vsphere.local\""}} 

  {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","messages":[{"args":[],"default_message":"Authentication required.","id":"com.vmware.vapi.endpoint.method.authentication.required"}],"challenge":"Basic realm=\"VAPI endpoint\",SIGN realm=94d0944913f0376e021aae6831a1807f5661b25f03aa889f3e709f1ef92f5b4b,service=\"VAPI endpoint\",sts=\"https://vcenter.sub1.domain.com/sts/STSService/vsphere.local\""}}

  
  
  
• The vCenter where the failed operation is performed share the same hostname with another vCenter of the VCF environment located in a different workload domain / subdomain.
  
  Example given:
  
  Workload Domain: vcenter.sub1.domain.com
  VI domain: vcenter.sub2.domain.com
  
  
• Running the following query on the SDDC manager on SSH give us multiple service accounts with the same username but belonging to different vCenters:
  
  psql -h localhost -U postgres -d platform -c "select c.username, vc.vm_hostname from credential c inner join vcenter vc on vc.id=c.entityid order by username;"
  
  Example given:
  
                       username                     |        vm_hostname         
--------------------------------------------------+----------------------------
 [email protected]                    | vcenter.sub1.domain.com
 [email protected]                    | vcenter.sub2.domain.com
 [email protected]                    | vcenter.sub1.domain.com
 [email protected]                    | vcenter.sub2.domain.com

Environment

VMware Cloud foundation 5.x

Cause

This is a known issue caused by service account naming convention that uses the SDDC hostname, vCenter hostname and the SSO domain to compose the username.
In the scenario were we got shared hostnames in the VCF environment the created service accounts will acquire the same name causing a conflict.

Resolution

A fix to address the issue should be provided in a future version of VCF.

Contact VCF Global support for a workaround.