Inventory sync of workload domains fails in VMware Aria Operations 9.x with vCenter API error 500

Products

• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

Inventory synchronization of workload domains within VMware Aria Operations 9.x fails. Reviewing the logs reveals that the API call to the relevant vCenter Server (api/vcenter/certificate-management/vcenter/trusted-root-chains) fails with a server-side 500 error.

In the vCenter Server /var/log/vmware/certificatemanagement/certificatemanagement-svcs.log, the following sequence of errors is observed, indicating the API runs but the session is not authenticated, failing when it attempts to use a SAML token for a locked service account (cms-<UNIQUE-ID>):

[tomcat-exec-5 [] INFO  com.vmware.certificatemanagement.vapi.impl.setup.AuthzPermissionValidator  opId=] User USER/ID invoked API com.vmware.vcenter.certificate_management.vcenter.trusted_root_chains.list
[tomcat-exec-5 [] INFO  com.vmware.certificatemanagement.vapi.impl.setup.AuthzPermissionValidator  opId=] AuthzClient created successfully.
[tomcat-exec-5 [] WARN  com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase  opId=] Asynchronous execution requested but no Executor configured. The request will be executed as synchronous one.
[tomcat-exec-5 [] INFO  com.vmware.certificatemanagement.vapi.impl.setup.AuthzServiceUtil  opId=] AuthzClient session not authenticated, performing loginBySamlToken
[tomcat-exec-5 [] INFO  com.vmware.certificatemanagement.vapi.impl.setup.AuthzServiceUtil  opId=] Is gateway: false
[tomcat-exec-5 [] INFO  com.vmware.certificatemanagement.vapi.impl.setup.AuthzServiceUtil  opId=] Generated ephemeral cert

[tomcat-exec-5 [] INFO  com.vmware.certificatemanagement.vapi.impl.setup.ServiceUtil  opId=] Acquiring SAML token for user cms-<UNIQUE-ID>@Domain
[tomcat-exec-5 [] ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: The account of the user trying to authenticate is locked. :: The account of the user trying to authenticate is locked. :: User account locked: {Name: cms-UNIQUE-ID, Domain: Domain} Please see the server log to find more detail regarding exact cause of the failure.

 

After six retries and failures due to the locked service account, the trusted-root-chains API fails with:

Could not validate permission information for operation com.vmware.vcenter.certificate_management.vcenter.trusted_root_chains.list invocation.
com.vmware.vim.binding.dataservice.fault.NotAuthenticatedFault: null 

Environment

 

• VMware Cloud Foundation 9.x
• VMware Aria Operations
• VMware vCenter Server

 

Cause

The trusted-root-chains API fails because the cms-<unique-id> service account is locked. The system attempts to acquire the SAML token for this user but is denied, causing the API call and subsequent inventory synchronization to fail.

Resolution

To resolve this issue and unlock the account, restart the certificate management service on the vCenter Server.

1. Connect to the affected vCenter Server via SSH.

2. Open a shell prompt.

3. Run the following command to restart the service:

   service-control --restart vmware-certificatemanagement
   

Once the service restarts, the account unlocks, and the API executes successfully. This allows the workload domain inventory synchronization to continue normally.