Rejected password events in vCenter Server from third party backup software

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server and ESXi host logs show multiple entries related to a 'Rejected password' error for a generic backup account. Backup jobs complete successfully despite these repeated authentication errors.
• vCenter Server and ESXi host logs display the following errors:
  
  /var/run/log/hostd.log
  
  

  YYYY-MM-DD:T:HH:MM:SS Er(163) Hostd[2098713]: [Originator@6876 sub=Default rhost=] [module:pam_lsass]pam_do_authenticate: error [login:<backup account@domain>][error code:2]
YYYY-MM-DD:T:HH:MM:SS Er(163) Hostd[2098713]: [Originator@6876 sub=Default rhost=] [module:pam_lsass]pam_sm_authenticate: failed [error code:2]
YYYY-MM-DD:T:HH:MM:SS Wa(164) Hostd[2098713]: [Originator@6876 sub=Vimsvc.HaSessionManager] Rejected password for user '<backup account@domain>' from [IP_ADDRESS] - session=####-####-####-####
YYYY-MM-DD:T:HH:MM:SS In(166) Hostd[2098713]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 53081 : Cannot login <backup account@domain>@[IP_ADDRESS]

  /var/log/vmware/vpxd/vpxd.log

  YYYY-MM-DDTHH:MM:SS.SSSZ | vAPI-I/O dispatcher-1 | [UUID] | [IP_ADDRESS] - - [[DATE]] "POST /rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags-on-objects HTTP/1.1" 200 12 "-" "RestSharp/[IP_ADDRESS]" 136
YYYY-MM-DDTHH:MM:SS.SSSZ | vAPI-I/O dispatcher-1 | [UUID] | [IP_ADDRESS] - - [[DATE]] "POST /rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags-on-objects HTTP/1.1" 200 12 "-" "RestSharp/[IP_ADDRESS]" 179

Environment

VMware vSphere 8.x

Cause

A generic account within the third party backup software attempts to authenticate against the vCenter Server or ESXi hosts using incorrect or outdated credentials.

Resolution

To resolve the issue, update the credentials for the affected backup service account:

1. Identify the source IP address of the failed login attempts from the vCenter Server or ESXi logs.
   • SSH into the ESXi host and run the following command to filter for rejected passwords events: cat /var/run/log/hostd.log | grep -i "Rejected password for user"
2. Locate the source machine using the IP address identified in step 2.
3. Update and re-enter the credentials for the affected service account on that machine.
4. If authentication errors persist in the logs after updating the credentials, engage the backup vendor to investigate application specific authentication triggers.