Error: Failed to load NSX Cluster from the Inventory during NSX Manager account password remediation - VMware Cloud Foundation

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation
• VMware NSX

Issue/Introduction

• Password remediation tasks for NSX Manager accounts initiated from the SDDC Manager or VCF OPS interface fail persistently.
  
  
• The SDDC Manager or VCF OPS UI displays the following error message during the remediation attempt:

  Failed to get NSX user details. The credentials were incorrect or the account specified has been locked.

• Direct SSH or Web UI authentication to the NSX-T Manager nodes utilizing the affected credentials succeeds, confirming no actual account lockout exists at the appliance level.
  
  
• The /var/log/vmware/vcf/operationsmanager/operationsmanager.log file on the SDDC Manager appliance records HTTP 403 Unauthorized exceptions during API invocation:

  Exception occurred during NSX API invocation
  java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) => {
      messages = [],
      data = struct => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
      errorType = UNAUTHORIZED
  }

• The /var/log/vmware/vcf/lcm/lcm-debug.log file on the SDDC Manager appliance records subsequent inventory load failures:

  Error auditing NSX Cluster vcf-nsx.example.com with exception {}
  com.vmware.evo.sddc.lcm.model.error.LcmException: Failed to load NSX Cluster from the Inventory
  Caused by: com.vmware.vapi.std.errors.Unauthorized: Unauthorized

• When checking details for the NSX Manager from platform db of SDDC Manager by connecting to it using SSH client and logging in as vcf user and switching user to root and executing command: psql -h localhost -d platform -U postgres -c "\x" -c "select status,version,cluster_fqdn,nsxt_cluster_details,configuration from nsxt;" , the system indicates an active status but fails the inventory audit with the following details:

  status               | ACTIVE
  version              | 4.2.3.3.0-25171318
  cluster_fqdn         | vcf-nsx.example.com
  ...
  configuration        | {
                       |   "vcfId": "446f##5-7c36-####-8937-19#####50dc",
                       |   "auditSucceeded": false,
                       |   "auditError": {
                       |     "errorCode": "Failed to load NSX Cluster from the Inventory",
                       |     "errorDetails": "error_message : Failed to load NSX Cluster from the Inventory, httpStatus : , error_code : 0"
                       |   },
                       |   "resourceId": "vcf-nsx.example.com",
                       |   "version": { "version": "4.2.3.3.0-25171318" }
                       | }

Environment

• VMware Cloud Foundation 5.x
• VMware Cloud Foundation 9.x

Cause

This condition occurs when the install-upgrade (upgrade-coordinator) service on the underlying NSX-T Manager nodes enters an unresponsive or stalled state. When SDDC Manager attempts to query the NSX-T API to validate the inventory and process the credential exchange, the stalled service incorrectly rejects the authentication payload with an HTTP 403 Forbidden response. This false rejection simulates a locked credential state, preventing SDDC Manager from successfully completing the password remediation workflow.

Resolution

To resolve this issue, the install-upgrade service must be restarted across all NSX Manager nodes within the affected cluster to restore standard API payload processing.

1. Establish an SSH connection or open a Web Console session to the first NSX Manager node for the impacted Domain.
2. Authenticate utilizing the admin credential of NSX Manager for that Domain.
3. Execute the following command to forcibly restart the upgrade-coordinator service: restart service install-upgrade
4. Repeat Steps 1 through 3 for all remaining nodes in the NSX-T Manager cluster.
5. Access the SDDC Manager interface and re-initiate the Password Remediation task for the affected NSX-T Manager accounts.
6. The task will now complete successfully and synchronize the credentials.