Unable to ssh into HCX Manager as admin

Products

• VMware HCX

Issue/Introduction

• On HCX managers, logging into the manager via ssh fails with an "Authentication Failure" instead of "Bad Username or Password".
• It is possible  to log in via the VM's console as root.
• In the /var/log/messages you see entries similar to the following:

messages:2025-11-10T01:10:00.089+00:00 fqdn.hcx.mgr sshd[4224]: Failed keyboard-interactive/pam for admin from ###.###.###.### port ### ssh2

2025-10-23T17:53:36.414+00:00 fqdn.hcx.mgrsshd[4120]: error: PAM: Authentication failure for admin from ###.###.###.##

• To confirm the issue you can enable DEBUG level logging in /etc/ssh/sshd_config, changing the LogLevel line from INFO to DEBUG

root [ /home/admin ]# cat /etc/ssh/sshd_config | grep syslog -A 2
SyslogFacility AUTHPRIV
LogLevel DEBUG

 

After changing the loglevel, when the issue occurs again, you should see a line similar to the following:

2025-11-03T22:28:24.902+00:00 HCX-Cloud sshd[11925]: pam_faillock(sshd:auth): Consecutive login failures for user admin account temporarily locked

Environment

HCX 4.10

HCX 4.11.1

HCX 4.11.2

HCX 4.11.3

Cause

The admin account has been locked.

Resolution

From the root console on the HCX Manager:

faillock --user admin --reset

This will clear the temporary lock.

Default Lockout time for admin is 24 hours. 

Default Lockout time for root is 5 minutes.

Additional Information

The issue is caused by an outside entity attempting to login with incorrect credentials. It is recommended to have your network evaluated for the IP's identified in the logs showing the failed login attempts to see if it is a security scan, or something more nefarious.