Unable to log into ESXi host via SSH: Server refused to start a Shell/command

Products

• VMware vSphere ESXi

Issue/Introduction

• Log into an ESXi host via SSH and see the client error 'Server refused to start a Shell/command'
• The following log messages in /var/log/auth.log. See Accessing DCUI/Console of ESXi using ALT+F Keys to check the auth.log in DCUI directly.
  
  In(38) sshd[]: Connection from <client-ipaddress> port ##
Er(83) sshd[]: PAM unable to dlopen(/lib/security/$ISA/pam_vmk_exec.so): libstdc++.so.6: failed to map segment from shared object
Er(83) sshd[]: PAM adding faulty module: /lib/security/$ISA/pam_vmk_exec.so
Er(35) sshd[]: error: fork(): Cannot allocate memory
Er(35) sshd[]: error: PAM: failed to start authentication thread: Cannot allocate memory

Environment

VMware vSphere ESXi 8.x

Cause

The SSH memory pool is exhausted.

Resolution

1. To check the memory configuration across the ESXi host.
   
   localcli --plugin-dir /usr/lib/vmware/esxcli/int sched group getmemconfig --group-path /host/vim/vimuser/terminal/ssh
   
   
2. Increase SSH pool memory allocation, eg, for the current max is 900MB to increase this to 1024MB. 
   
   localcli --plugin-dir /usr/lib/vmware/esxcli/int sched group setmemconfig --group-path /host/vim/vimuser/terminal/ssh  --max <1024> --units mb
   
   
3. Check if the SSH access gets through for the target ESXi host.
   If the issue still persists, kindly contact Broadcom support.