Unable to login with admin account after resetting the password on the NSX manager appliance

Products

• VMware NSX

Issue/Introduction

• When resetting a local user password following the steps in the Admin guide - resetting passwords on an NSX appliance, the passwords are not being synchronized among the NSX Managers in the cluster.
• You made sure you issued the command "touch /var/vmware/nsx/reset_cluster_credentials" but despite that, password is not synchronized.
• Rolling reboot of the NSX managers does not resolve the issue.
• Only the SSH and GUI login with admin password is impacted, the root login works fine on the NSX manager appliances.
• NSX manager admin password is not expired.
  
• We could see indexing errors under /var/log/search/search-manager.log.
  

  <timestamp>  WARN UfoIndexer-BatchExecutor-search_manager-2 UfoIndexingServiceImpl 5945 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] [Indexing: JsonDocumentGeneration] Data to index is null for UfoObject{operationType=INDEX, descriptor=IndexingTypeDescriptor{tableName='VmPropertiesMsg', streamTag=MANAGER}, identifier=uuid {
    left: <left_decimal_ID>
    right: <right_decimal_ID>
  }, timestamp.sequence =897480125, timestamp.epoch=48}

Environment

VMware NSX 4.2.x

Cause

The failure to synchronize passwords is caused by a service-level deadlock or indexing inconsistency within the nsx-mp-api-server and the Search Manager component. If the API server does not correctly register the credential reset trigger, the metadata synchronization service fails to propagate the change, leading to "null" data entries in the search index as seen in the logs.

Resolution

Use below steps to reset the password of the NSX manager appliance:

1. SSH to NSX Managers as root.
2. Stop the nsx-mp-api-server service:
   /etc/init.d/nsx-mp-api-server stop
3. To reset the password for admin, run the command
   passwd admin
4. Run the command 
   touch /var/vmware/nsx/reset_cluster_credentials
5. Restart the nsx-mp-api-server service:
   /etc/init.d/nsx-mp-api-server start
6. Now, re-attempt login to the NSX manager using admin via SSH and GUI.

Note: If unable to login with new password, then run Resync command from the Active NSX manager running the cluster VIP:
# su admin -c start search resync all

Additional Information

NSX credentials are not being synchronized between NSX Managers after manual password reset.