Error : "A problem has occurred on the server. Please retry or contact the service provider and provide the reference token" during NSX Manager upgrade in VCF Operations

Products

• VCF Operations

Issue/Introduction

• Attempting to upgrade the VMware NSX Manager to version 9.1.0.0 via VCF operations fails with the following error in the UI:
  
  Error: A problem has occurred on the server. Please retry or contact the service provider and provide the reference token

• The following error messages are observed in the failed workflow's subtask:
  
  
• • Error: Unable to create the service account on <NSX_FQDN>. Error: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => { messages = [], data = => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services}, errorType = UNAUTHORIZED, challenge = <null> }

• • Error: NSX Audit failed for <NSX_FQDN> with error error_message :
Failed to load NSX Cluster from the Inventory, httpStatus : , error_code : 0

• Constant inventory synchronization error messages are observed in the task list of the VCF operations:

Environment

VMware Cloud Foundation (VCF) 9.1

Cause

This issue is caused by an out-of-band authentication mismatch. The admin, root, and audit passwords were updated locally on the NSX Manager appliance, causing them to fall out of sync with the credential store. Because VCF Operations continuously polls the NSX Manager using the stale credentials, it triggers an account lockout resulting in a 403 Unauthorized error.

Resolution

1. Utilize the lookup_passwords command-line utility to retrieve the expected legacy passwords currently stored in the database.
   Reference: Use the SDDC Manager Appliance to Look Up Account Credentials

2. Access the primary NSX Manager node via SSH and revert the local passwords to match the database. This action will clear the active account lockouts.
   Reference: Credential operations fail on NSX-T Components in SDDC Manager

3. Navigate to Security -> Password Management in the SDDC Manager UI. For each affected account, click the vertical ellipsis (⋮), select "Update Password" and enter the credentials retrieved from the database in Step 1.

4. Execute the "Apply All" action under Configuration Upgrades, which will successfully provision the required service account.
   
   Example:
   
   

5. Retry the NSX upgrade once the service account is created and API access is fully restored.