ESXi root account locked with error "Access denied"
search cancel

ESXi root account locked with error "Access denied"

book

Article ID: 312772

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  1. Remote access for ESXi local user account 'root' has been locked for 900 seconds after X failed login attempts.
  2. Cannot login to ESXi Host via Host Client UI
  3. Connecting to ESXi Host via ssh results with "Access denied"
  4. Rebooting the ESXi Host will allow access immediately but issue persists again after sometime.
  5. Only way to connect to ESXi Host is via DCUI.
  6. Reviewing logs collected via vCenter server shows below snippet.
/var/run/log/vobd.log
YYYY-MM-DDTHH:MM:SS.972Z: [UserLevelCorrelator] 1478941347002us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 52 failed login attempts.
YYYY-MM-DDTHH:MM:SS.972Z: [UserLevelCorrelator] 1478941347243us: [esx.audit.account.locked]Remote access for ESXi local user account 'root' has been locked for 900 seconds after 52 failed login attempts.
YYYY-MM-DDTHH:MM:SS.972Z: [GenericCorrelator]   1478941347002us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 52 failed login attempts.

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue typically arises due to third-party monitoring software or scanning applications that are attempting to connect to the ESXi host using incorrect credentials. These applications might be:

  • Misconfigured with outdated or incorrect login credentials.
  • Performing automated scans without proper authorization.

Resolution

Steps to identify the system connecting to ESXi Host with incorrect credentials using the log files mentioned below.
 

/var/run/log/auth.log
YYYY-MM-DDTHH:MM:SSZ sshd[318439]: Connection from 10.xx.yy.zz port 62430
YYYY-MM-DDTHH:MM:SSZ sshd[318439]: Accepted keyboard-interactive/pam for root from 10.xx.yy.zz port 62430 ssh2
YYYY-MM-DDTHH:MM:SSZ sshd[335145]: Connection from 10.xx.yy.zz port 61574
YYYY-MM-DDTHH:MM:SSZ sshd[335148]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.xx.yy.zz  user=root
YYYY-MM-DDTHH:MM:SSZ sshd[335145]: error: PAM: Authentication failure for root from 10.xx.yy.zz
YYYY-MM-DDTHH:MM:SSZ sshd[335145]: Accepted keyboard-interactive/pam for root from 10.xx.yy.zz port 61574 ssh2
 
/var/run/log/rhttpproxy.log
YYYY-MM-DDTHH:MM:SS7Z    verbose rhttpproxy[278CB70] [Originator@6876 sub=Proxy Req 14800] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x02103358] _serverNamespace = /vpxa action = Allow _port = 8089
YYYY-MM-DDTHH:MM:SS7Z    verbose rhttpproxy[29C2B70] [Originator@6876 sub=Proxy Req 15643] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x02103358] _serverNamespace = /vpxa action = Allow _port = 8089
YYYY-MM-DDTHH:MM:SS6Z    verbose rhttpproxy[2606B70] [Originator@6876 sub=Proxy Req 15721] New proxy client <SSL(<io_obj p:0x02303ba8, h:16, <TCP '10.xx.yy.zz:443'>, <TCP '10.xx.yy.zz:63789'>>)>
YYYY-MM-DDTHH:MM:SS7Z    verbose rhttpproxy[27CDB70] [Originator@6876 sub=Proxy Req 15721] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x02103358] _serverNamespace = /vpxa action = Allow _port = 8089



Temporarily unlock the account:

To workaround this issue, please follow the below steps:

  1. Login to ESXi host directly via console (for Dell its iDRAC, for HPE its iLO, for Cisco its CIMC etc) Direct Console Interface (DCUI) with "root" account.
  2. Press "F2", go to "Troubleshooting Options".
  3. Enable "ESXi Shell" using "Enable ESXi Shell" Option and go back to main menu.
  4. Press "CTRL+ALT+F1" to switch to "ESXi Shell"
  5. Type Command "pam_tally2 --user root" to view total failed login attempt from root. 
Example "there are 60 failed login attempt."

# pam_tally2 --user root
Login               Failures Latest failure     From
root                60    10/09/20 15:59:56    unknown
  1. Type Command "pam_tally2 --user root --reset" to reset the lock and validate if failed attempts are reset.
Example

# pam_tally2 --user root --reset
Login              Failures Latest failure     From
root               61    10/09/20 15:60:56    unknown

# pam_tally2 --user root
Login           Failures Latest failure       From
root                0
  1. Now we will be able to login to ESXi via ESXi Host Client UI or via ssh.

Additional Information

ESXi Account Lockout Behavior
Account locking is supported for access through SSH and through the vSphere Web Services SDK. The Direct Console Interface (DCUI) and the ESXi Shell do not support account lockout. By default, a maximum of five failed attempts is allowed before the account is locked. The account is unlocked after 15 minutes by default.

Configuring Login Behavior
You can configure the login behavior for your ESXi host with the following advanced options:
Security.AccountLockFailures. Maximum number of failed login attempts before a user's account is locked. Zero disables account locking.
Security.AccountUnlockTime. Number of seconds that a user is locked out.
Security.PasswordHistory. Number of passwords to remember for each user. Zero disables password history.

Reference: ESXi Passwords and Account Lockout

Powered by Wolken Software