SDDC password rotation causing SSH service to be stopped on NSX manager node holding VIP

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

SSH service gets stopped in NSX manager node associated with cluster VIP when password rotation task is performed from SDDC.

Environment

VMware NSX
VMware NSX-T Data Center

Cause

Command to check NSX manager cluster VIP

nsx-mngr> get cluster vip

The SSH disablement is as per the design. Password management would disable SSH after password operations.

Password management does the following :

1. Enables SSH
2. Updates the password (tests the password updated or rotated using SSH)
3. Disables SSH

auth.log :
**********
x.x.x.x = SDDC IP

log$ less auth.log* | grep -i "pam_unix" | grep -i ssh
2024-03-07T04:03:20.933155+00:00 sshd 27372 - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=root
2024-03-07T05:05:10.288344+00:00 sshd 10531 - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=root

proxy log :
************
NSX is receiving POST API /api/v1/node/services/ssh?action=stop from SDDC IP x.x.x.x as part of the design workflow.

proxy$ less localhost_access_log* | grep -i "/api/v1/node/services/ssh?action=stop"
2024-03-07T04:03:24.513Z x.x.x.x - "POST /api/v1/node/services/ssh?action=stop HTTP/1.1" 200 160 1629 1629
2024-03-07T05:05:14.126Z x.x.x.x - "POST /api/v1/node/services/ssh?action=stop HTTP/1.1" 200 160 1888 1887

SDDC log :
*************

2024-03-07T04:03:11.169+0000 DEBUG [0000000000000000,0000] [c.v.v.p.helper.NsxtApiUtil,Thread-501] Status from stop SSH service API for NSXT of vip-nsx-mgmt.vrack.vsphere.local is : 200 OK

Resolution

Versions where this is an expected behavior : VCF 4.4.x, 4.5.x, 5.0.x.

Version where this is fixed : VCF 5.1.0 onwards. The password rotation will not disable SSH for VCF 5.1.0 and onwards.

Workaround :

Restart SSH service manually

root@nsx-mngr:~# service ssh restart