SDDC Manager add cluster fails at Obtain vSphere Lifecycle Manager Cluster Image Version with 'Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server'

Products

• VMware vCenter Server
• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Adding a cluster to a VI domain fails at the task: Obtain vSphere Lifecycle Manager Cluster Image Version.
• The SDDC Manager UI displays the following error:

  Message: Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server vCenter.example.com
  Remediation Message: Make sure that vCenter Server vcenter.example.com contains the relevant depot base image for the vSphere Lifecycle Manager Cluster Image

• In /var/log/vmware/vcf/domainmanager/domainmanager.log on the SDDC Manager, you see unauthenticated errors:

  DEBUG [vcf_dm,,################################,,fc09] [c.v.v.c.f.a.GetPersonalityFullVersionAction,dm-exec-4]  Getting personality full version for a personality with base image display name ESXi and base image display version 8.0 U3f - 24784735.
  INFO  [vcf_dm,,################################,,fc09] [c.v.vcf.vapi.vsphere.VapiSession,dm-exec-4]  Renew VAPI session - acquiring new vapi session.
  INFO  [vcf_dm,,################################,,fc09] [c.v.vcf.vapi.vsphere.VapiRestClient,dm-exec-4]  VapiRestClient creating new session to vCenter vCenter.example.com with user [e-mail]...

   

  ERROR [vcf_dm,,################################,,fc09] [c.v.v.v.v.VCAPIExploreResponseErrorHandler,dm-exec-4]  URL: https://example/rest/com/vmware/cis/session, HttpMethod: POST, ResponseBody: {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","challenge":"SIGN realm=\"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:#\",sts=\"https://vCenter.example.com/sts/STSService/vsphere.local\", Basic realm=\"vCenter\"","messages":[]}}
  ERROR [vcf_dm,################################,fc09] [c.v.v.v.v.VCAPIExploreResponseErrorHandler,dm-exec-4]  Unauthenticated Error: Error (com.vmware.vapi.std.errors.error) =>

• In /var/log/vmware/sso/vmware-identity-sts.log on the vCenter Server, you see invalid credentials:

  ERROR sts[44:tomcat-http--6] [CorId==########-####-####-####-############]] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldap://vCenter.example.com:389] because [Invalid credentials] therefore will not attempt to use any secondary URIs
  WARN sts[44:tomcat-http--6] [CorId==########-####-####-####-############]] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Failed to authenticate using SRP binding
  com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

Environment

• VMware Cloud Foundation
• VMware SDDC Manager

Cause

This issue occurs if the service account password was rotated between the time the cluster creation task was initially started and the time a re-try attempt was initiated. The cached or workflow-specific credentials no longer match the active credentials.

Resolution

To resolve this issue, retrieve the newly rotated service account credentials and update the workflow spec file.

Follow the below steps

1. Take an offline snapshot of the SDDC Manager VM
2. Retrieve the current service account credentials from SDDC Manager by following the steps in KB 327195.
3. Update the service account password in the active workflow spec file by following the instructions in KB 314620.
4. Return to the SDDC Manager UI and re-try the failed cluster addition job.