vCenter Server down with services unable to start
search cancel

vCenter Server down with services unable to start

book

Article ID: 405456

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

vpxd-svcs service fails to attempt to start

vpxd-svcs.log: 
 :566) [commons-pool2-2.12.0.jar:2.12.0]
         at org.apache.commons.pool2.impl.GenericObjectPool.addObject(GenericObjectPool.java:222) [commons-pool2-2.12.0.jar:2.12.0]
         at com.vmware.cis.server.util.impl.InitPoolTask.run(InitPoolTask.java:44) [inventory-server.jar:?]
         at java.base/java.lang.Thread.run(Unknown Source) [?:?]
 2025-07-22T18:05:17.342Z [Thread-11 [] INFO  com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor  opId=] Provided credentials are not valid.
 2025-07-22T18:05:17.342Z [Thread-11 [] WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 15
 2025-07-22T18:05:37.370Z [Thread-11 [] ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP fault
 com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
         at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:163) ~[jaxws-rt-2.3.4.jar:2.3.4]
         at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:98) ~[jaxws-rt-2.3.4.jar:2.3.4]
        at com.sun.xml.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:244) ~[jaxws-rt-2.3.4.jar:2.3.

trustmanagement-svc.log: 
2025-07-22T03:59:06.802Z [inventoryPermissionConverterScheduler-1 [] ERROR com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter  opId=] VPXD AuthZ inventory permission conversion failed
com.vmware.svcaccount.token.exceptions.AcquireTokenException: SAML token request was rejected
        at com.vmware.svcaccount.token.TokenClient.acquireTokenForSvcAccount(TokenClient.java:192) ~[svcaccountlib.jar:?]
        at com.vmware.svcaccount.token.TokenClient.acquireHokToken(TokenClient.java:144) ~[svcaccountlib.jar:?]
        at com.vmware.vcenter.trustmanagement.vapi.impl.setup.ServiceUtil.getAuthenticatedSsoAdminClient(ServiceUtil.java:244) ~[libservice.jar:?]
        at com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter.getCurrentSsoDomains(InventoryPermissionConverter.java:120) ~[libservice.jar:?]
        at com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter.convertAliasPermissions(InventoryPermissionConverter.java:91) [libservice.jar:?]
        at jdk.internal.reflect.GeneratedMethodAccessor285.invoke(Unknown Source) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.3.42.jar:5.3.42]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.3.42.jar:5.3.42]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

ldif collected from vCenter looks like this: 

dn: cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
objectClass: vmwSTSIdentityStore
objectClass: top
cn: vsphere.local
vmwSTSAlias: SYSTEM-DOMAIN  <------ Legacy Configuration 
vmwSTSAuthenticationType: SRP
vmwSTSConnectionStrings: ldap:/vcenter_fqdn:389
vmwSTSDomainName: vsphere.local
vmwSTSDomainType: SYSTEM_DOMAIN
vmwSTSGroupBaseDN: DC=vsphere,DC=local
vmwSTSProviderType: IDENTITY_STORE_TYPE_VMWARE_DIRECTORY
vmwSTSServiceUseMachineAccount: false
vmwSTSTimeout: 0
vmwSTSUpnSuffixes: SYSTEM-DOMAIN  <------Legacy Configuration
vmwSTSUserBaseDN: DC=vsphere,DC=local

Environment

VMware vCenter Server 

Resolution

1. Take powered down snapshots of all linked vCenters. 

2. Download Jxplorer and Java Using JXplorer to connect to the vSphere Single Sign-on

3. Navigate to Services > IdentityManager > Tenants > vsphere.local > Identity Providers > vsphere.local and remove both configurations (vmwSTSUpnSuffixes: SYSTEM-DOMAIN and vmwSTSAlias: SYSTEM-DOMAIN) as these are legacy configurations. 

4. Restart services on vCenter: service-control --stop --all && service-control --start --all

Additional Information

Powered by Wolken Software