Password update, rotate, remediate failing intermittently for NSX Edge node accounts and sometimes SDDC Manager UI falsely shows the NSX Edge accounts as Disconnected.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

Password rotation for NSX-T Manager node from SDDC Manager fails with TEST_AFTER_UPDATE error.

error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

yyyy-mm-ddT hh:mm:ss. INFO  [vcf_om,07bff5d6a16a4eb0,1db3] [c.v.v.p.r.CancelPasswordTransactionHandler,http-nio-127.0.0.1-7300-exec-2] Cancelling password rotate for entity..{"transactionId":2006,"entityName":"<nsxt_manager FQDN>","entityId":"<entity_id>","oldPassword":"*****","newPassword":"*****","entityType":"NSXT_MANAGER","credentialType":"*****","transactionStatus":"FAILED","transactionTime":" ","updateStage":"TEST_AFTER_UPDATE","workflowId":"<workflow_id>","username":"root","diagnosticMessage":"{\"errorCode\":\"PASSWORD_UPDATE_LOGIN_FAILED_POST_PASSWORD_CHANGE\",\"arguments\":[\"*****\"],\"errorMessage\":\"Unable to login using new password for entity <nsxt_manager FQDN>, after password change.\",\"updateStage\":\"*****\",\"referenceToken\":\"<token>\"}"}

 

Environment

VMware Cloud Foundation 4.5

VMware Cloud Foundation 5.x

Cause

Fetching ssh status for NSX-T Manager is getting times out.
Current ssh status polling is done 12 times at an interval of 5 seconds which makes the total time to 1 min.

Resolution

Workaround:

Increase the time (max 7 mins) for the ssh status to be updated.

Steps:

1. Take snapshot of SDDC Manager VM
2. SSH to SDDC Manager with vcf user and su to root
3. append the ssh status retry attempt in application-prod.properties file
   

   echo "password.nsxt_edge.ssh_status_check.retry_attempts=85" >> /etc/vmware/vcf/operationsmanager/application-prod.properties

4. Restart Operationsmanager service
   

   systemctl restart operationsmanager

   
   
   
5. Retry password rotation from SDDC Manager