Root password remediate operation for ESXi host fails with an error as, PASSWORD_MANAGER_VALIDATE_ESXI_CREDENTIALS_FAILED, Cannot complete login due to an incorrect user name or password
search cancel

Root password remediate operation for ESXi host fails with an error as, PASSWORD_MANAGER_VALIDATE_ESXI_CREDENTIALS_FAILED, Cannot complete login due to an incorrect user name or password

book

Article ID: 401305

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  •  Below log entries we see in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

yyyy-mm-ddThh:mm.075+0000 ERROR [vcf_om,684a744c49f3e6e081347a758da8dae5,ac0c] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-13] Cannot complete login due to incorrect credentials: esxifqdn, svc-vcf-esxi.
yyyy-mm-ddThh:mm.075+0000 ERROR [vcf_om,684a744c49f3e6e081347a758da8dae5,ac0c] [c.v.v.p.h.EsxiHostCommandExecutor,om-exec-13] Exception occured in getting connection to ESXi host : esxifqdn using a connection via: svc-vcf-esxi, {}
java.util.concurrent.ExecutionException: (vim.fault.InvalidLogin) {
   faultCause = null,
   faultMessage = null
}
        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereClient.<init>(VsphereClient.java:121)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.connect(VcManagerBase.java:514)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:495)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerBase.<init>(VcManagerBase.java:468)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.getVcManagerBase(VcManagerFactory.java:436)
        at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerFactory.java:52)
        at com.vmware.vcf.passwordmanager.helper.EsxiHostCommandExecutor.testPasswordViaLogin(EsxiHostCommandExecutor.java:309)
        at com.vmware.vcf.passwordmanager.update.changers.EsxiChanger.doTest(EsxiChanger.java:161)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:432)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:201)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:100)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:88)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at com.vmware.vcf.common.tracing.TraceRunnable.run(TraceRunnable.java:59)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

 

  • On ESXi, we see below entries in,

     /var/run/log/hostd.log 

yyyy-mm-ddThh:mm.069Z In(166) Hostd[2101760]: [Originator@6876 sub=Vimsvc opID=093fb0f8 sid=523ab8d4] [Auth]: User svc-vcf-esxi
yyyy-mm-ddThh:mm.069Z Wa(164) Hostd[2101760]: [Originator@6876 sub=Vimsvc opID=093fb0f8 sid=523ab8d4] Refresh function is not configured.User data can't be added to scheduler.User name: svc-vcf-esxi
yyyy-mm-ddThh:mm.069Z In(166) Hostd[2101760]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=093fb0f8 sid=523ab8d4] Event 552 : Cannot login user [email protected]: no permission
yyyy-mm-ddThh:mm.002Z Er(163) Hostd[2101766]: [Originator@6876 sub=VMkernelStatsProvider(000000e324fa9030)] GetKernelStatValues: Detected error while retrieving stats: VSINode(2652): Not found (status=195887107)
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101774]: [Originator@6876 sub=Solo.Vmomi] Activation finished; <<523ab8d4-7a05-70dd-dfe9-42c12faa126a, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 42982'>>, ha-sessionmgr, vim.SessionManager.login, <vim.version.v7_0, internal, 7.0.0.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x000000e326411598]>
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101774]: [Originator@6876 sub=Solo.Vmomi] Arg userName:
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101737]: --> "svc-vcf-esxi"
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101774]: [Originator@6876 sub=Solo.Vmomi] Arg password:
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101737]: --> (not shown)
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101737]: -->
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101774]: [Originator@6876 sub=Solo.Vmomi] Arg locale:
yyyy-mm-ddThh:mm.070Z Db(167) Hostd[2101737]: --> (null)
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101774]: [Originator@6876 sub=Solo.Vmomi] Throw vim.fault.
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101774]: [Originator@6876 sub=Solo.Vmomi] Result:
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101737]: --> (vim.fault.NoPermission) {
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101737]: -->    object = 'vim.Folder:ha-folder-root',
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101737]: -->    privilegeId = "System.View",
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101737]: -->    msg = "",
yyyy-mm-ddThh:mm.070Z In(166) Hostd[2101737]: --> }
yyyy-mm-ddThh:mm.081Z In(166) Hostd[2101745]: [Originator@6876 sub=Solo.Vmomi opID=093fb0fe sid=5293d731] Activation finished; <<5293d731-2b13-8a7f-569d-4984683f33aa, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 42982'>>, ha-sessionmgr, vim.SessionManager.logout, <vim.version.v7_0, internal, 7.0.0.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x000000e2f0860b08]>
yyyy-mm-ddThh:mm.081Z In(166) Hostd[2101745]: [Originator@6876 sub=Solo.Vmomi opID=093fb0fe sid=5293d731] Throw vim.fault.NotAuthenticated
yyyy-mm-ddThh:mm.081Z In(166) Hostd[2101745]: [Originator@6876 sub=Solo.Vmomi opID=093fb0fe sid=5293d731] Result:
yyyy-mm-ddThh:mm.081Z In(166) Hostd[2101737]: --> (vim.fault.NotAuthenticated) {
yyyy-mm-ddThh:mm.081Z In(166) Hostd[2101737]: -->    object = 'vim.SessionManager:ha-sessionmgr',
yyyy-mm-ddThh:mm.081Z In(166) Hostd[2101737]: -->    privilegeId = "System.View",

 

     syslog.log

yyyy-mm-ddThh:mm.948Z Er(83) sshd-session[2689572]: pam_access(sshd:account): access denied for user svc-vcf-esxi' from ip_address'
yyyy-mm-ddThh:mm.956Z In(14) addvob[2689586]: Log for VMware ESXi version=8.0.3 build=build-24674464 option=Release
yyyy-mm-ddThh:mm.956Z In(14) addvob[2689586]: Could not expand environment variable HOME.
yyyy-mm-ddThh:mm.956Z In(14) addvob[2689586]: Could not expand environment variable HOME.
yyyy-mm-ddThh:mm.956Z In(14) addvob[2689586]: Using VMware ESXi syslog APIs
yyyy-mm-ddThh:mm.960Z Er(35) sshd-session[2689475]: error: PAM: User account has expired for svc-vcf-esxi from ip_address
yyyy-mm-ddThh:mm.964Z Er(35) sshd-session[2689475]: error: Received disconnect from ip_address port 64410:14: No supported authentication methods available [preauth]
yyyy-mm-ddThh:mm.964Z In(38) sshd-session[2689475]: Disconnected from authenticating user svc-vcf-esxi ip_address port 64410 [preauth]

Environment

VCF 5.2.x

SDDC 5.2.x

Cause

  • The password for the svc-vcf-esxi account was unavailable.
  • Resetted the svc-vcf-esxi password directly on the ESXi host but failed to restore SSH access. The service account was missing Administrator privileges on the target ESXi host.

 

 

 

Resolution

  1. Take a snapshot of the SDDC Manager VM.

     2.  Align the svc-vcf-esxi account password with the ESXi root password by performing the following steps.

     3. Identify the ESXi host ID in the database:

          psql -h localhost -U postgres -d platform -c "select id,hostname from host where hostname='esxifqdn';"

     4. Using the ESXi host ID obtained in the previous step, retrieve the associated credentials:

          psql -h localhost -U postgres -d platform -c "select * from credential where entityid='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';"

     5. The command returns two credential entries: one for the root account and one for the svc-vcf-esxi account.

     6. Note the credential ID associated with the svc-vcf-esxi account.

     7. If the secret values for the root and svc-vcf-esxi accounts differ, copy the secret value from the root account and update it for the svc-vcf-esxi account using the following command:

       psql -h localhost -U postgres -d platform -c "update credential set secret='secret_copied_from_root' where id='id_collected_in_step_6';"

     8. On the ESXi host, reset the svc-vcf-esxi account password to match the root password using the passwd command.

     9. Set the required permissions for the svc-vcf-esxi account:

         esxcli system permission set --id svc-vcf-esxi -r Admin

     10. Verify that SSH access using the svc-vcf-esxi account is successful.

     11. Retry the ESXi password remediation workflow.

 

Additional Information

  • Check if ESXi svc-vcf-esxi account is present on ESXi using the below command:

esxcli system account list

  • In case, the account is not present you can add the svc account and set the password and Admin permissions.
Powered by Wolken Software