NSX Manager Upgrade Fails with error "NSX_T_MANAGER resourceId: nsx-manager.example.com status changed to SKIPPED COMPLETED_WITH_FAILURE"

Products

• VMware SDDC Manager

Issue/Introduction

During an upgrade of NSX Manager from version 4.2.0 to 4.2.1.3 on VMware Cloud Foundation (VCF) 5.2.1.1 running on VxRail 8.0.300, the upgrade process failed with the error:
"NSX_T_MANAGER resourceId: nsx-manager.example.com status changed to SKIPPED COMPLETED_WITH_FAILURE".

The NSX Manager upgrade was skipped due to an authentication failure during pre-check validation. Logs indicated a 401 Unauthorized HTTP request from VxManager, despite confirmed validity of the vCenter management account credentials when tested manually.

Environment

SDDC manager 5.2.1.1

Cause

This is caused by a mismatch or outdated encrypted credentials for the vCenter management account stored in VxRail Manager. While the credentials were functionally valid, their encrypted form in VxRail did not align with the current password, leading to authentication failures during upgrade pre-checks. 

Resolution

1. Rotate and Update vCenter Management Account Password:

   • Change the password for the vCenter management account through the SDDC UI.
    

2. Update Encrypted Credentials in VxRail Manager:

   • Dell should have a process to do this
   • Ensure the new password is encrypted using the same method as previously stored in VxRail Manager.
    

3. Restart Services in VxRail Manager:

   • Execute the following commands to restart relevant services:
     systemctl restart vmware-marvin
systemctl restart runjars
      
    

4. Reattempt NSX Manager Upgrade:

   • Initiate the NSX Manager upgrade process again through SDDC manager.

Additional Information

Verify that the updated credentials have correct permissions in vCenter.

Check logs (lcm-debug.log) post-resolution to confirm no remaining authentication errors.