Remediation of ESXi root account shows error "vim.fault.InvalidLogin" on SDDC manager

Products

• VMware SDDC Manager

Issue/Introduction

• ESXi hosts(s) root and service accounts show disconnected on the SDDC Manager Password Management page:

• Remediating the root account from SDDC manager fails with "vim.fault.InvalidLogin" on operationsmanager.log of the SDDC Manager as observed below :
  

  /var/log/vmware/vcf/operationsmanager/operationsmanager.log
  
  YYYY-MM-DDThh:mm:ssDEBUG [vcf_om,6####8,2##1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec
  -2#] Update operation started asynchronously
  ...
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,6####8,2##1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec
  -2#] About to do testBeforeRemediation step..
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,6####8,2##1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec
  -2#] Remediation required: Performing test before remediation for username: root
  ...
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,6####8,2##1] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-2#] Connecting to https://<esxi_fqdn>:443/sdk
  ...
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,6####8,2##1] [c.v.e.s.c.c.v.vsphere.VsphereClient,om-exec-2#] Failed to connect to https://<esxi_fqdn>:443/sdk
  ...
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,6####8,2##1] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-2#] Cannot complete login due to incorrect credentials: <esxi_fqdn>, svc-vcf-<esxi_host>.
  ...
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,6####8,2##1] [c.v.e.s.c.c.v.vsphere.VsphereClient,om-exec-2#] Failed to connect to https://<esxi_fqdn>:443/sdk
  YYYY-MM-DDThh:mm:ss WARN  [vcf_om,6####8,2##1] [c.v.e.s.c.c.v.vsphere.VsphereClient,om-exec-2#] Error logging out of session
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,6####8,2##1] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-2#] Cannot complete login due to incorrect credentials: <esxi_fqdn>, svc-vcf-<esxi_host>.
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,6####8,2##1] [c.v.v.p.h.EsxiHostCommandExecutor,om-exec-2#] Exception occured in getting connection to ESXi host : <esxi_fqdn> using a connection via: svc-vcf-<esxi_host>, {} java.util.concurrent.ExecutionException: (vim.fault. InvalidLogin)
  YYYY-MM-DDThh:mm:ss ERROR [vcf_om,6####8,2##1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-2#] (vim.fault.InvalidLogin) {
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,6####8,2##1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-2#] Error Message : (vim.fault.InvalidLogin) {
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,6####8,2##1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-2#] About to mark resource state as error...
  YYYY-MM-DDThh:mm:ss DEBUG [vcf_om,6####8,2##1] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-2#] Password operation failed for root

• Service accounts are found to be missing from the ESXi host(s) affected
• Host profile was applied on ESXi host(s) recently which can be further verified from vpxd.log of vCenter server as shown below:
  

  /var/log/vmware/vpxd/vpxd.log
  
  YYYY-MM-DDThhmm:ss info vpxd[06##7] [Originator@6876 sub=moHostProfile opID=3#####9][CheckCompliance]: Host <esxi_fqdn> is not compliant with profile Host Profile:
  (vim.profile.ComplianceResult.ComplianceFailure) [
  --> (vim.profile.ComplianceResult.ComplianceFailure) {
  -->   failureType =
  "com.vmware.vim.profile.Profile.security.UserAccountProfile.UserAccountProfile.ComplianceError.UserNotInProfile.label",
  -->   message = "User svc-vcf-<esxi_host> not present in profile."
  --> },
  --> (vim.profile.ComplianceResult.ComplianceFailure) {
  -->   failureType =
  "com.vmware.vim.profile.Profile.security.UserAccountProfile.UserAccountProfile.ComplianceError.PasswordMismatch.label",
  -->   message = "Password in profile does not match that on host for root."
  --> }
  ]

• At the same time, hostd.log on the concerned ESXi host shows removal of service account:
  

  /var/run/log/hostd.log
  
  YYYY-MM-DDThh:mm:ss In(1#6) Hostd[6####5]: Event 2###5: Account svc-vcf-<esxi_host> was removed on host :<esxi_fqdn>

Environment

VMware Cloud Foundation 5.x

Cause

This issue happens when a host profile is applied on the ESXi host(s) despite the compliance errors indicating the service account removal in a VCF environment. For root remediation, service account needs to be in a Connected state.

Resolution

To resolve the issue, recreate the service accounts on affected ESXi host(s) by following Re-create missing SDDC Manager Service account on an ESXi host.