Adding a cluster to VI domain fails at Obtain vSphere Lifecycle Manager Cluster Image Version with 'Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server'
search cancel

Adding a cluster to VI domain fails at Obtain vSphere Lifecycle Manager Cluster Image Version with 'Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server'

book

Article ID: 409515

calendar_today

Updated On:

Products

VMware vCenter Server VMware Cloud Foundation

Issue/Introduction

Adding a cluster to VI domain fails at Obtain vSphere Lifecycle Manager Cluster Image Version with 'Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server'.

Error Stack:

Message: Failed to get vSphere Lifecycle Manager Cluster Image full version from vCenter Server vCenter.example.com

Remediation Message: Make sure that vCenter Server vCenter.example.com contains the relevant depot base image for the vSphere Lifecycle Manager Cluster Image

Reference Token: ######

Cause: I/O error on POST request for "<a href="https://vCenter.example.com/rest/com/vmware/cis/session" :"="" title="https://vCenter.example.com/rest/com/vmware/cis/session" style="box-sizing: border-box; scrollbar-width: thin; scrollbar-color: rgb(128, 210, 242) rgba(0, 0, 0, 0); word-break: break-word; position: relative !important; max-width: 100%; float: none !important; cursor: pointer; color: blue; text-decoration: underline;">https://vCenter.example.com/rest/com/vmware/cis/session": {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","challenge":"SIGN realm=\"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:#\",sts=\"<a href="https://vCenter.example.com/sts/STSService/vsphere.local/" "="" title="https://vCenter.example.com/sts/stsservice/vsphere.local/" style="box-sizing: border-box; scrollbar-width: thin; scrollbar-color: rgb(128, 210, 242) rgba(0, 0, 0, 0); word-break: break-word; position: relative !important; max-width: 100%; float: none !important; cursor: pointer; color: blue; text-decoration: underline;">https://vCenter.example.com/sts/STSService/vsphere.local\", Basic realm=\"vCenter\"","messages":[]}} {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","challenge":"SIGN realm=\"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:#\",sts=\"<a href="https://vCenter.example.com/sts/STSService/vsphere.local/" "="" title="https://vCenter.example.com/sts/stsservice/vsphere.local/" style="box-sizing: border-box; scrollbar-width: thin; scrollbar-color: rgb(128, 210, 242) rgba(0, 0, 0, 0); word-break: break-word; position: relative !important; max-width: 100%; float: none !important; cursor: pointer; color: blue; text-decoration: underline;">https://vCenter.example.com/sts/STSService/vsphere.local\", Basic realm=\"vCenter\"","messages":[]}}

 

/var/log/vmware/vcf/domainmanager/domainmanager.log 

YYYY-MM-DDTHH:MM:SS.227+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.v.c.f.a.GetPersonalityFullVersionAction,dm-exec-4]  Getting personality full version for a personality with base image display name ESXi and base image display version 8.0 U3f - 24784735.
YYYY-MM-DDTHH:MM:SS.231+0000 INFO  [vcf_dm,,################################,,fc09] [c.v.vcf.vapi.vsphere.VapiSession,dm-exec-4]  Renew VAPI session - acquiring new vapi session.
YYYY-MM-DDTHH:MM:SS.231+0000 INFO  [vcf_dm,,################################,,fc09] [c.v.vcf.vapi.vsphere.VapiRestClient,dm-exec-4]  VapiRestClient creating new session to vCenter vCenter.example.com with user [email protected]...
YYYY-MM-DDTHH:MM:SS.231+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.vcf.vapi.vsphere.VapiRestClient,dm-exec-4]  Executing REST request: Type POST, URL https://vCenter.example.com/rest/com/vmware/cis/session
YYYY-MM-DDTHH:MM:SS.231+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.v.s.h.VcfRestTemplateHttpClient,dm-exec-4]  Http request connect timeout milliseconds 180000
YYYY-MM-DDTHH:MM:SS.231+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.v.s.h.VcfRestTemplateHttpClient,dm-exec-4]  Http request read timeout milliseconds 180000
YYYY-MM-DDTHH:MM:SS.233+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,dm-exec-4]  Security config retrieved {"fipsMode":false}
YYYY-MM-DDTHH:MM:SS.245+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.v.s.t.DynamicTrustManager,dm-exec-4]  Checking validity of certificate chain CN=vCenter.example.com,  
YYYY-MM-DDTHH:MM:SS.245+0000 DEBUG [vcf_dm,,################################,,fc09] [c.v.v.s.t.DynamicTrustManager,dm-exec-4]  Certificate chain CN=vCenter.example.com, 
YYYY-MM-DDTHH:MM:SS.280+0000 ERROR [vcf_dm,,################################,,fc09] [c.v.v.v.v.VCAPIExploreResponseErrorHandler,dm-exec-4]  URL: https://example/rest/com/vmware/cis/session, HttpMethod: POST, ResponseBody: {"type":"com.vmware.vapi.std.errors.unauthenticated","value":{"error_type":"UNAUTHENTICATED","challenge":"SIGN realm=\"##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:#\",sts=\"https://vCenter.example.com/sts/STSService/vsphere.local\", Basic realm=\"vCenter\"","messages":[]}}
2025-08-25T15:27:09.280+0000 ERROR [vcf_dm,################################,fc09] [c.v.v.v.v.VCAPIExploreResponseErrorHandler,dm-exec-4]  Unauthenticated Error: Error (com.vmware.vapi.std.errors.error) =>

 

/var/log/vmware/sso/vmware-identity-sts.log 


YYYY-MM-DDTHH:MM:SS.423Z INFO sts[44:tomcat-http--6] [CorId=########-####-####-####-############] [com.vmware.identity.sts.impl.STSImpl] Entering issue() token...
YYYY-MM-DDTHH:MM:SS.429Z WARN sts[44:tomcat-http--6] [CorId==########-####-####-####-############]] [com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 49
YYYY-MM-DDTHH:MM:SS.430Z WARN sts[44:tomcat-http--6] [CorId==########-####-####-####-############]] [com.vmware.identity.idm.server.ServerUtils] cannot bind connection: [ldap://vCenter.example.com:389, [email protected]]
YYYY-MM-DDTHH:MM:SS.430Z ERROR sts[44:tomcat-http--6] [CorId==########-####-####-####-############]] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldap://vCenter.example.com:389] because [Invalid credentials] therefore will not attempt to use any secondary URIs
YYYY-MM-DDTHH:MM:SS.430Z WARN sts[44:tomcat-http--6] [CorId==########-####-####-####-############]] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Failed to authenticate using SRP binding
com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

 

/var/log/vmware/vmdird/vmdird.log

YYYY-MM-DDTHH:MM:SS.429Z:t@140118895597120:ERROR: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

 

Environment

VMware Cloud Foundation 5.X

Cause

Issue is caused due to the fact that the service account password has been rotated between the time when the cluster creation task had been initiated and a re-try attempt.

Resolution

  1. Take a snapshot of the SDDC manager VM.
  2. Retrieve the service accounts credentials from SDDC Manager following the steps mentioned in in the KB Retrieve the service accounts credentials from SDDC Manager
  3. Update the service account password in the workflow spec file following the steps mentioned in the KB Re-try an existing workflow by modifying the workflow spec file
  4. Re-try the job
Powered by Wolken Software