ERROR: Remote access for ESXi local user account 'root' has been locked for 120 seconds

Products

VMware vSphere ESXi

Issue/Introduction

Remote access to an ESXi host fails for the local root user despite valid credentials.
Rebooting the host temporarily resolves the issue but does not prevent recurrence.
The account is locked for 120 seconds following a large number of failed login attempts, as indicated in log entries from /var/run/log/vobd.log.

/var/run/log/vobd.log
YYYY-MM-DDTHH:MM:SSZ: [GenericCorrelator] 1222053089769us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 120 seconds after 44090 failed login attempts.
YYYY-MM-DDTHH:MM:SSZ: [UserLevelCorrelator] 1222053089978us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 120 seconds after 44090 failed login attempts.
YYYY-MM-DDTHH:MM:SSZ: [UserLevelCorrelator] 1222061520175us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 120 seconds after 44091 failed login attempts.
YYYY-MM-DDTHH:MM:SSZ: [GenericCorrelator] 1222061520175us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 120 seconds after 44091 failed login attempts.

Environment

6.x
7.x
8.x

Cause

The lockout is triggered by repeated authentication failures originating from a third-party service or tool.
This behavior may be observed in environments with Nutanix clusters, where the CVM (Customer Virtual Machine) service was identified as the source of excessive login attempts.

Resolution

Identifying and preventing the source of the logins is the recommended action to resolve the issue, the <IP Address> can be identified via the logging of hostd.log as shown below:

/var/run/log/hostd.log

YYYY-MM-DDTHH:MM:SSZ Wa(164) Hostd[2098850]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=######## sid=########] Rejected password for user root from <IP Address> - session=########-####-####-####-############
YYYY-MM-DDTHH:MM:SSZ Wa(164) Hostd[2098880]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=######## sid=########] Rejected password for user root from <IP Address> - session=########-####-####-####-############
YYYY-MM-DDTHH:MM:SSZ Wa(164) Hostd[2098856]: [Originator@6876 sub=Vimsvc.HaSessionManager opID=######## sid=########] Rejected password for user root from <IP Address> - session=########-####-####-####-############

Additionally syslog.log and rhttpproxy.log located at /var/run/log/ can also be compared to gain further information on the logins

Workaround:

Temporarily Prevent Account Lockout by changing the following Advanced Configuration:

Navigate to Configuration > Advanced Options in the ESXi host settings.
Set Security.AccountLockFailures to 0 to disable account lockouts after failed login attempts.

Nutanix Cluster-Specific Steps (if applicable):
Follow this Procedure of Action (POA):

On all hosts, set Security.AccountLockFailures to 0.
Stop the Genesis cluster service on the Nutanix CVM using the command.
```
allssh genesis stop
```
Unlock the root account via console access on each host:
```
pam_tally2 -u root -r
```
Run the Nutanix CVM script to update SSH keys:
```
/usr/local/nutanix/cluster/bin/fix_host_ssh
```
Restart the Genesis cluster service:
```
cluster restart_genesis
```
Restore Security.AccountLockFailures to its original/default value (typically 5)