SDDC password rotation causing SSH service to be stopped on NSX manager node holding VIP

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

SSH service gets stopped in NSX manager node associated with cluster VIP when password rotation task is performed from SDDC.

Environment

• VMware NSX
• VMware NSX-T Data Center

Cause

Command to check NSX manager cluster VIP

nsx-mngr> get cluster vip

The SSH disablement is as per the design. Password management would disable SSH after password operations.

Password management does the following :

1. Enables SSH
2. Updates the password (tests the password updated or rotated using SSH)
3. Disables SSH

auth.log :
**********
#.#.#.# = SDDC IP

log$ less auth.log* | grep -i "pam_unix" | grep -i ssh
YYYY-MM-DDTHH:MM:SS+Z sshd 27372 - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=#.#.#.# user=root
YYYY-MM-DDTHH:MM:SS+Z sshd 10531 - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=#.#.#.# user=root

proxy log :
************
NSX is receiving POST API /api/v1/node/services/ssh?action=stop from SDDC IP x.x.x.x as part of the design workflow.

proxy$ less localhost_access_log* | grep -i "/api/v1/node/services/ssh?action=stop"
YYYY-MM-DDTHH:MM:SS+Z #.#.#.# - "POST /api/v1/node/services/ssh?action=stop HTTP/1.1" 200 160 1629 1629
YYYY-MM-DDTHH:MM:SS+Z #.#.#.# - "POST /api/v1/node/services/ssh?action=stop HTTP/1.1" 200 160 1888 1887

SDDC log :
*************

YYYY-MM-DDTHH:MM:SS+Z DEBUG [0000000000000000,0000] [c.v.v.p.helper.NsxtApiUtil,Thread-501] Status from stop SSH service API for NSXT of domain is : 200 OK

Resolution

• Versions where this is an expected behavior : VCF 4.4.x, 4.5.x, 5.0.x.
• Version where this is fixed : VCF 5.1.0 onwards. The password rotation will not disable SSH for VCF 5.1.0 and onwards.

Workaround :

• Restart SSH service manually on NSX manager by running the command : 
  
  

  root@nsx-mngr:~# service ssh restart

Additional Information

Managing Local User Accounts : Managing Local User Accounts
For information related to NSX local node users, please refer KB - Error while executing command for user audit in NSX Manager CLI