Unable to SSH into vCenter as the connection is abruptly terminated

Products

• VMware vCenter Server

Issue/Introduction

• Unable to SSH into vCenter from any SSH client
• SSH client connection is reset before the root password can be entered.
• Issue is seen both on root and SSO logins via SSH.
• Confirmed that root and SSO accounts are valid by logging into VAMI and the vCenter VM remote console.
• The below snippets are observed during a failed SSH login

/var/log/audit/sshinfo.log
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: reprocess config line 134: Deprecated option RhostsRSAAuthentication
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: PAM: initializing for "root"
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: PAM: setting PAM_RHOST to "###.###.#.#"
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: PAM: setting PAM_TTY to "ssh"
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: userauth_send_banner: sent [preauth]
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: monitor_read_log: child log fd closed
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: do_cleanup
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: PAM: cleanup
YYYY-MM-DDTHH:MM:SS vCenter_fqdn sshd[34622]: debug1: Killing privsep child 34623

From the vCenter VM remote console, check journal logs using: journalctl -xe
MMM DD HH:MM:SS vCenter_fqdn audit[26774]: SECCOMP auid=4294967295 uid=50 gid=50 ses=4294967295 subj=unconfined pid=26774 comm="sshd" exe="/usr/sbin/sshd" sig=31 arch=c000003e syscall=230 compat=0 ip=0x7f849d24eaf3 code=0x0
MMM DD HH:MM:SS vCenter_fqdn audit[26774]: ANOM_ABEND auid=4294967295 uid=50 gid=50 ses=4294967295 subj=unconfined pid=26774 comm="sshd" exe="/usr/sbin/sshd" sig=31 res=1
MMM DD HH:MM:SS vCenter_fqdn sshd[26773]: debug1: userauth_send_banner: sent [preauth]
MMM DD HH:MM:SS vCenter_fqdn sshd[26773]: debug1: monitor_read_log: child log fd closed
MMM DD HH:MM:SS vCenter_fqdn sshd[26773]: debug1: do_cleanup
MMM DD HH:MM:SS vCenter_fqdn sshd[26773]: debug1: PAM: cleanup
MMM DD HH:MM:SS vCenter_fqdn sshd[26773]: debug1: Killing privsep child 26774

Environment

VMware vCenter Server 7.x

Cause

This was caused as the glibc photon 3 package in vCenter was manually updated with a photon 5 package of a incompatible version.

Resolution

Broadcom does not support any modifications or customizations to the underlying operating system and packages included in a VMware-branded virtual appliance.
Please refer: VMware Virtual Appliances and customizations to operating system and included packages

It is strongly recommended to revert back to a valid snapshot or restore vCenter from a VAMI or a VM backup. If valid backups do not exist, then take a valid snapshot of the vCenter VM and then follow the steps below to reload the correct glibc version.

tdnf install glibc-2.28-24.ph3
systemctl daemon-reload 
systemctl restart sshd

Validate if the server shows the correct glibc RPM's using:
rpm -qa | grep -i glibc

Reboot the vCenter server to apply the changes and validate if SSH connectivity is working on the vCenter. 

Note:
If the RPM reload does not work and no valid backups exist, then its recommended to redeploy the vCenter server.