Generating a Certificate Signing Request (CSR) for NSX Manager fails in VMware Cloud Foundation with error: "403 Forbidden: The credentials were incorrect or the account specified has been locked"

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

 

• Attempting to generate a Certificate Signing Request (CSR) for an NSX Manager instance via the SDDC Manager / VCF Operations UI fails.

• The SDDC Manager user interface displays the following error message:

  Failed to generate CSR for <nsx-fqdn> due to: 403 Forbidden: "{"module_name":"common-services","error_message":"The credentials were incorrect or the account specified has been locked.","error_code":403}".
  

• Review the SDDC Manager Operations Manager logs (e.g., /var/log/vmware/vcf/operationsmanager/operationsmanager.log or /var/log/vmware/vcf/vcf_om/vcf_om.log) to confirm the API connection is rejected by NSX Manager. The logs will explicitly identify the locked service account being passed during the authentication attempt:

   

  DEBUG [vcf_om,...] [c.v.v.c.f.p.n.s.s.CredentialMgmtServiceImpl,...] Fetching service account credentials for NSX <REDACTED_SECRET>
  DEBUG [vcf_om,...] [c.v.e.s.c.s.s.ServiceCredentialsHelper,...] Getting credentials for target type NSXT_MANAGER, entity ID <REDACTED_SECRET> and service type SDDC_MANAGER
  DEBUG [vcf_om,...] [c.v.v.c.n.s.c.c.ApiConnection,...] Creating ApiClient to https://<REDACTED_HOSTNAME>:443 with username <REDACTED_SECRET>
  DEBUG [vcf_om,...] [c.v.v.c.n.s.c.c.ApiConnection,...] Created ApiClient connection to: <REDACTED_HOSTNAME>
  INFO  [vcf_om,...] [o.b.jsse.provider.ProvTlsClient,...] [client #2040 @1296f1a] opening connection to <REDACTED_HOSTNAME>:443
  INFO  [vcf_om,...] [o.b.jsse.provider.ProvTlsClient,...] [client #2040 @1296f1a] established connection with <REDACTED_HOSTNAME>:443
  ERROR [vcf_om,...] [c.v.v.c.n.s.c.c.ComplexHelpers,...] Exception occurred during NSX API invocation
  java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.Unauthorized: Unauthorized (com.vmware.vapi.std.errors.unauthorized) (statusCode:403) => {
      messages = [],
      data =  => {error_message=The credentials were incorrect or the account specified has been locked., error_code=403, module_name=common-services},
      errorType = UNAUTHORIZED
  }
        at com.vmware.vcf.common.nsxt.sdk.client.connection.NodeVersionOperations.getVersion(NodeVersionOperations.java:30)
        at com.vmware.vcf.common.nsxt.sdk.client.connection.ApiConnection.preferPolicyOverMp(ApiConnection.java:560)
        at com.vmware.vcf.common.nsxt.sdk.client.connection.ApiConnection.initializeShim(ApiConnection.java:537

   

 

Environment

VMware Cloud Foundation (VCF) 9.x

Cause

The internal SDDC Manager service account utilized to authenticate against the target NSX Manager API has become locked out or desynchronized. Certificate management operations utilize these credentials, causing the NSX REST API to immediately reject the connection and abort the CSR generation.

Resolution

Warning: This resolution requires modifying the internal SDDC Manager PostgreSQL database and deleting user accounts via the NSX Manager REST API. Take a file-based backup or snapshot of the SDDC Manager appliance and NSX Manager cluster before proceeding.

To resolve this issue, you must purge the locked service account from both the SDDC Manager database and the NSX Manager local user store, and then allow SDDC Manager to regenerate a synchronized account:

Step 1: Identify and Remove the Stale Credential from SDDC Manager

1. Connect to the SDDC Manager appliance via SSH using the vcf user account, then elevate to root:

   Bash

    

   su -
   

2. Connect to the SDDC Manager PostgreSQL platform database:

   Bash

    

   psql -U postgres -d platform -h localhost
   

3. Locate the locked service account username associated with the NSX Manager instance:

   SQL

    

   SELECT entity_type, entity_id, username, status, error_message FROM passwordmanager.credential_expiry WHERE entity_type = 'NSXT_MANAGER' AND status = 'FAILED';
   

4. Verify the exact username in the credential table (replace <svc_acct> with the username identified in Step 3, e.g., svc-alvcf-mddr01-sddc-alvcf-9798):

   SQL

    

   SELECT * FROM credential WHERE username = '<svc_acct>';
   

5. Delete the service account record from the SDDC Manager credential table:

   SQL

    

   DELETE FROM credential WHERE username = '<svc_acct>';
   

6. Exit the PostgreSQL database prompt:

   SQL

    

   \q
   

Step 2: Delete the Service Account from NSX Manager

1. From the SDDC Manager command line (or any machine with network access to NSX Manager), query the NSX Manager API to obtain the internal User ID (node_user_id) of the locked service account:

   Bash

    

   curl -ks https://<NSXmanager_FQDN>/api/v1/node/users -u 'admin:<ADMIN_PASSWORD>' | grep -B 2 -A 5 "<svc_acct>"
   

   (Note: You can also locate the numerical User ID by logging into the NSX Manager UI and navigating to System > User Management > Local Users).

2. Delete the user account directly from NSX Manager via the REST API (replace <USER_ID> with the numerical ID located in the previous step):

   Bash

    

   curl -ks https://<NSXmanager_FQDN>/api/v1/node/users/<USER_ID> -u 'admin:<ADMIN_PASSWORD>' -X DELETE
   

Step 3: Regenerate the Service Account via SDDC Manager

1. Log in to the SDDC Manager user interface.

2. Navigate to Inventory > Workload Domains and select the Workload Domain containing the affected NSX Manager instance.

3. Click Services > Remediate (or execute the Config Drift workflow depending on your specific VCF release).

4. SDDC Manager will detect that the NSX Manager service account is missing from its inventory and will automatically recreate and register a new, synchronized service account on the target NSX Manager.

5. Monitor the task in the SDDC Manager UI until the remediation completes successfully.

6. Navigate back to Security > Certificate Management and re-initiate the NSX Manager CSR generation workflow.