Workload domain creation from SDDC Manager fails at subtask 'Add ESXi host to vSphere Cluster'

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• Error in SDDC Manager UI

  Message: Hosts cannot be added to the cluster domain-c9
  Cause: Retriable operation 'Adding host esxi01.example.com to cluster domain-c9' failed to complete after 6 retries. Error getting thumbprint for host esxi01.example.com (vim.fault.InvalidLogin) { faultCause = null, faultMessage = null } Cannot complete login due to an incorrect user
  name or password.

   

• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log in SDDC Manager

  ERROR [vcf_dm,6818##############9006,3282] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-17]  Failed to connect to https://esxi01.example.com:443/sdk
  java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
     faultCause = null,
     faultMessage = null
  }
          at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
  ..
  Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.
          at jdk.internal.reflect.GeneratedConstructorAccessor2803.newInstance(Unknown Source)
  ..
          ... 26 common frames omitted
  WARN  [vcf_dm,6818##############9006,3282] [c.v.e.s.c.c.v.vsphere.VsphereClient,dm-exec-17]  Error logging out of session
  com.vmware.vim.binding.vim.fault.NotAuthenticated: The session is not authenticated.
  ..
  ERROR [vcf_dm,6818##############9006,3282] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-17]  Cannot complete login due to incorrect credentials: esxi01.example.com, svc-vcf-esxi01.
  ERROR [vcf_dm,6818##############9006,3282] [c.v.v.c.f.p.a.i.AddHostsToClusterContractAction,dm-exec-17]  Error getting thumbprint for host esxi01.example.com
  java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
     faultCause = null,
     faultMessage = null

   

• Error in /var/run/log/hostd.log in ESXi host

  Wa(164) Hostd[######]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=89####5f sid=52####eb] Rejected password for user svc-vcf-esxi01 from <SDDC_Manager_IP> - session=528e####-####-####-########c05b
  In(166) Hostd[######]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=89####5f sid=52####eb] Event 1581 : Cannot login svc-vcf-esxi01@<SDDC_Manager_IP>
  In(166) Hostd[######]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<528e####-####-####-########c05b, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 38940'>>, ha-sessionmgr, vim.SessionManager.login,
  <vim.version.v7_0, internal, 7.0.0.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x00000051d8fb37e8]>
  Db(167) Hostd[######]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
  Db(167) Hostd[2241279]: --> "svc-vcf-esxi01"
  Db(167) Hostd[######]: [Originator@6876 sub=Solo.Vmomi] Arg password:
  Db(167) Hostd[2241279]: --> (not shown)
  Db(167) Hostd[2241279]: -->
  Db(167) Hostd[######]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
  Db(167) Hostd[2241279]: --> (null)
  In(166) Hostd[######]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.InvalidLogin
  In(166) Hostd[######]: [Originator@6876 sub=Solo.Vmomi] Result:

Environment

VMware Cloud Foundation 5.x

Cause

Password for ESXi service account (e.g. svc-vcf-esxi01) was changed after getting created and was updated to SDDC. Add Domain task still try to use the initial password of the account.

Resolution

1. Find the ESXi service account passwords stored in SDDC Manager database - Refer Retrieve the service accounts credentials from SDDC Manager 
2. Reset the password obtained from Step # 1
   1. SSH to ESXi host with root
   2. Run the below command

      esxcli system account set --id=<svc-vcf-esxi_sgort_name> --password=<password retrieved from Step # 1>

3. Remediate the same password in SDDC Manager
4. Retry the Workload domain creation in SDDC Manager