ESXi SSH Error "No supported authentication methods available (server sent: publickey,keyboard-interactive)"

Products

• VMware vSphere ESXi

Issue/Introduction

•   When trying to SSH into an ESXi host, a TTY error is thrown in the client being used to query for the SSH connection:

"No supported authentication methods available (server sent: publickey,keyboard-interactive)"

•   When consoled into the ESXi host via the DCUI, standard commands like esxcli or esxtop fail to run, citing memory issues.

Example:

[root@esxi01:~]  esxtop

Out of memory

 

• Rebooting the ESXI host provides a temporary fix.

• In vmkernel.log, you see where processes ask for memory, but there is not enough memory available to satisfy the ask.

YYYY-MM-DDTHH:MM:SS.000Z In(187) vmkernel: cpu14:24715883)uw.24715883(000000000) requires 132 KB, asked 132 KB from ssh (760) which has 819128 KB occupied and 72 KB available.

• In /var/log/vobd.log you see a large number of SSH sessions being opened, but not a directly corresponding number of closed sessions:

• • 6 sessions opened, but only 3 closed. This is typically followed by another session creation chain that has disproportionally low session closures.
  • NOTE: SSH Sessions will continue after SSH access is disabled. The remaining 3 opened from the example below will remain open until closed.

YYYY-MM-DDTHH:MM:SS.162Z In(14) vobd[2098147]:  [GenericCorrelator] 4179181034553us: [vob.user.ssh.session.opened] SSH session was opened for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.162Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181034553us: [vob.user.ssh.session.opened] SSH session was opened for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.162Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181034775us: [esx.audit.ssh.session.opened] SSH session was opened for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.367Z In(14) vobd[2098147]:  [GenericCorrelator] 4179181239116us: [vob.user.ssh.session.opened] SSH session was opened for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.367Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181239116us: [vob.user.ssh.session.opened] SSH session was opened for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.367Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181239334us: [esx.audit.ssh.session.opened] SSH session was opened for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.621Z In(14) vobd[2098147]:  [GenericCorrelator] 4179181493812us: [vob.user.ssh.session.closed] SSH session was closed for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.621Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181493812us: [vob.user.ssh.session.closed] SSH session was closed for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.622Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181493996us: [esx.audit.ssh.session.closed] SSH session was closed for 'example_user@#.#.#.#'.
YYYY-MM-DDTHH:MM:SS.683Z In(14) vobd[2098147]:  [GenericCorrelator] 4179181555512us: [vob.user.ssh.disabled] SSH access has been disabled
YYYY-MM-DDTHH:MM:SS.683Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181555512us: [vob.user.ssh.disabled] SSH access has been disabled
YYYY-MM-DDTHH:MM:SS.683Z In(14) vobd[2098147]:  [UserLevelCorrelator] 4179181555679us: [esx.audit.ssh.disabled] SSH access has been disabled.

Environment

VMware vSphere ESXi 8.x

Cause

The symptoms point to a potential memory leak, or something causing new processes to not get the memory allocated needed to run.

 

Resolution

Workaround:

• Reboot the host.
• If problem persists, contact Broadcom Technical Support to help identify memory leaks on the host.
• If the host is a VXRail host, please reach out to Dell support.