From the SDDC Manager, password rotation for NSX Manager and Edge nodes is failing.

Products

• VMware NSX

Issue/Introduction

• In a VCF environment, the SDDC Manager can be configured to frequently rotate the passwords of NSX Manager and Edge nodes. For more details, please refer - Rotate Passwords.
  
  
• Password rotation fails with the tasks timing out. SDDC Manager logs complain about the NSX Manager being inaccessible.
  
  
• Accessing the NSX Manager webpage using the VIP may fail with the error - "An error occurred during OAuth2 operation. Please contact your administrator to resolve the issue. { "error": "invalid_request", "error_description": "Must provide a matching redirect URI." }".
  
  
• In case of the above error, there are two possibilities:
  • The primary NSX Manager (the one holding the VIP) is inaccessible.
  • We may be encountering the issue documented in the KB - Login to NSX-T with vIDM user fails with error 'Must provide a matching redirect uri'.
    
    
• To skip the vIDM authentication and locally login, we can use the URL "https://<nsx-mgr-vip-fqdn/ip>/login.jsp?local=true".
  
  
• If the primary NSX Manager is having issues, even local login using the VIP may not work. In such a case, try to directly access the NSX Manager nodes using the URL "https://<nsx-mgr-fqdn/ip>/login.jsp?local=true".
  
  
• If only the primary NSX Manager is experiencing issues, one of the direct logins may not be able to load all the pages of NSX UI, may complain about object not found, metadata error, indexing failure, etc. The other two Manager nodes may be correctly displaying all the pages and objects.

Environment

VMware NSX

Resolution

• Take an SSH session to all the three Manager nodes, run the command "get cluster status" to check the status of all the services from the perspective of the Cluster Boot Manager (CBM).
• Run the command "get cluster vip" to know which Manager node is the leader.
• If any of the services are down on the leader node or, to isolate if the issue is with only the leader node, re-assign the VIP to one of the other nodes using the steps from the KB - How to change the owner of the NSX Manager Cluster VIP.
• Alternatively, rebooting the leader node too will get the VIP moved to the other nodes.
• Post reboot or, reassignment of the VIP, try to access the NSX Manager UI using the VIP and check if it works.
• Try to directly access the rebooted NSX Manager webpage and check if the UI now is able to load all the pages and objects.
• Should the problem persist even after the reboot, there can be other issues like resource (cpu/mem/storage) constraints, service failures, etc. which, will need further review. At this point, please raise a ticket with the Broadcom Support Team via the Broadcom Support Portal.