SDDC Manager password remediation for vCenter root password fails with error: Failed executing command in VM with FQDN <vCenter_FQDN>, error : Failed to authenticate with the guest operating system using the supplied credentials.

Products

• VMware SDDC Manager
• VMware Cloud Foundation 4.x

Issue/Introduction

• SDDC Manager password remediation for vCenter root password fails below error in log file /var/log/vmware/vcf/operationsmanager/operationsmanager.log 
  
  com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Failed executing command in VM with FQDN <vCenter FQDN>, error : Failed to authenticate with the guest operating system using the supplied credentials. at com.vmware.vcf.passwordmanager.update.changers.VcenterRootUserChanger.doTest(VcenterRootUserChanger.java:76) ERROR [vcf_om,e35ece7b3eac706f,63e6] [c.v.v.p.u.c.VcenterRootUserChanger,om-exec-22] Exception in doTest of com.vmware.vcf.passwordmanager.update.changers.VcenterRootUserChanger

• VC root password is matching with what is stored in SDDC Manager lookup_passwords and ssh connection to VC is working with the same credentials.

• Below steps to create a test program on vCenter Server MOB using Guest Operation Manager doesn't return any PID
  • Turn on MOB: https://knowledge.broadcom.com/external/article?articleId=340046
  • Find out the VM MoId of vCenter Server VM using VC MOB:
    • Browse https://<vCenter FQDN>/mob/?moid=SearchIndex&method=findByDnsName
    • Enter below values, replace the <vCenter FQDN> with the FQDN of vCenter Server:
datacenter = "" (leave it blank)
dnsName = <vCenter FQDN>,
vmSearch = true
    • Click Invoke Method and it will show the VM MoID.
  • Run a command using the Guest Operations Manager to confirm the PID:
    • Browse https://<vCenter FQDN>/mob/?moid=guestOperationsProcessManager&method=startProgram
    • Enter the following details to test the program execution (enter the VC VM MoId, username as root and the root password:
      
        vm =
    <vm type="VirtualMachine">VM_MoId_from_above_step</vm>

  auth =
   <auth xsi:type="internalvim25:NamePasswordAuthentication">
   <interactiveSession>false</interactiveSession>
   <username>root</username>
   <password>******</password>
   </auth>
  spec =
   <spec>
   <programPath>/bin/uname</programPath>
   <arguments>-a</arguments>
   </spec>
    • Click Invoke Method. Note down the Return value i.e. the process id. (e.g. 12345)
    • If the above steps fails to generate a PID, it indicates that there might be a problem with vCenter Server guest operations and steps in this KB will help to resolve the issue.

Resolution

• Take snapshot of vCenter Server or take a backup
• SSH to vCenter Server
• Change directory to /etc/pam.d/
  
  cd /etc/pam.d/
  
  
• Backup /etc/pam.d/vmtoolsd
  
  cp vmtoolsd vmtoolsd.pre_sshd_pre_mgmt


• Replace /etc/pam.d/vmtoolsd with /etc/pam.d/sshd.pre_mgmt
  
  cp sshd.pre_mgmt vmtoolsd


• Restart vgauth and vmtoolsd
  
  systemctl restart vgauthd vmtoolsd


• Check the service status and confirm it is active/running
  
systemctl --no-pager status vgauthd vmtoolsd


• Retry VC root password remediation from SDDC Manager