The vCenter Server is inaccessible because the vpxd-svcs service failed to initiate
search cancel

The vCenter Server is inaccessible because the vpxd-svcs service failed to initiate

book

Article ID: 405456

calendar_today

Updated On:

Products

VMware SDDC Manager VMware vCenter Server 8.0

Issue/Introduction

vpxd-svcs service failed to start with following errors from vpxd-svcs.log logs.

  • /var/log/vmware/vpxd-svcs/vpxd-svcs.log
 :566) [commons-pool2-2.12.0.jar:2.12.0]
         at org.apache.commons.pool2.impl.GenericObjectPool.addObject(GenericObjectPool.java:222) [commons-pool2-2.12.0.jar:2.12.0]
         at com.vmware.cis.server.util.impl.InitPoolTask.run(InitPoolTask.java:44) [inventory-server.jar:?]
         at java.base/java.lang.Thread.run(Unknown Source) [?:?]
 <YYYY-MM-DD>T<time> [Thread-11 [] INFO  com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor  opId=] Provided credentials are not valid.
 <YYYY-MM-DD>T<time> [Thread-11 [] WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 15
 <YYYY-MM-DD>T<time> [Thread-11 [] ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP fault
 com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
         at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:163) ~[jaxws-rt-2.3.4.jar:2.3.4]
         at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:98) ~[jaxws-rt-2.3.4.jar:2.3.4]
        at com.sun.xml.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:244) ~[jaxws-rt-2.3.4.jar:2.3.

 

  • /var/log/vmware/trustmanagement/trustmanagement-svcs.log
<YYYY-MM-DD>T<time> [inventoryPermissionConverterScheduler-1 [] ERROR com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter  opId=] VPXD AuthZ inventory permission conversion failed
com.vmware.svcaccount.token.exceptions.AcquireTokenException: SAML token request was rejected
        at com.vmware.svcaccount.token.TokenClient.acquireTokenForSvcAccount(TokenClient.java:192) ~[svcaccountlib.jar:?]
        at com.vmware.svcaccount.token.TokenClient.acquireHokToken(TokenClient.java:144) ~[svcaccountlib.jar:?]
        at com.vmware.vcenter.trustmanagement.vapi.impl.setup.ServiceUtil.getAuthenticatedSsoAdminClient(ServiceUtil.java:244) ~[libservice.jar:?]
        at com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter.getCurrentSsoDomains(InventoryPermissionConverter.java:120) ~[libservice.jar:?]
        at com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter.convertAliasPermissions(InventoryPermissionConverter.java:91) [libservice.jar:?]
        at jdk.internal.reflect.GeneratedMethodAccessor285.invoke(Unknown Source) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.3.42.jar:5.3.42]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.3.42.jar:5.3.42]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
  • /var/log/vmware/applmgmt/applmgmt.log
<YYYY-MM-DD>T<time> [3179]INFO:vmware.appliance.backup_restore.schedule_impl:Schedule with id 'default' updated successfully.
<YYYY-MM-DD>T<time> [3179]ERROR:vmware.appliance.extensions.authentication.authentication_sso:Unhandled exception during SAML token validation
Traceback (most recent call last):
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate
    self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in validate_certificate
    self.add_x509_pem_header(c)) for c in certsFromToken]
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in <listcomp>
    self.add_x509_pem_header(c)) for c in certsFromToken]
  File "/usr/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1825, in load_certificate
    _raise_current_error()
  File "/usr/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.crypto.Error: []
<YYYY-MM-DD>T<time> [3179]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
Traceback (most recent call last):
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 507, in validate
    self.validate_certificate()
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in validate_certificate
    self.add_x509_pem_header(c)) for c in certsFromToken]
  File "/usr/lib/applmgmt/lib/extensions/py/vmware/appliance/extensions/authentication/authentication_sso.py", line 674, in <listcomp>
    self.add_x509_pem_header(c)) for c in certsFromToken]
  File "/usr/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1825, in load_certificate
    _raise_current_error()
  File "/usr/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.crypto.Error: []

 

ldif collected from vCenter looks like this: 

dn: cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
objectClass: vmwSTSIdentityStore
objectClass: top
cn: vsphere.local
vmwSTSAlias: SYSTEM-DOMAIN  <------ Legacy Configuration 
vmwSTSAuthenticationType: SRP
vmwSTSConnectionStrings: ldap:/vcenter_fqdn:389
vmwSTSDomainName: vsphere.local
vmwSTSDomainType: SYSTEM_DOMAIN
vmwSTSGroupBaseDN: DC=vsphere,DC=local
vmwSTSProviderType: IDENTITY_STORE_TYPE_VMWARE_DIRECTORY
vmwSTSServiceUseMachineAccount: false
vmwSTSTimeout: 0
vmwSTSUpnSuffixes: SYSTEM-DOMAIN  <------Legacy Configuration
vmwSTSUserBaseDN: DC=vsphere,DC=local

Environment

VMware vCenter Server 8.X

Cause

Issue occurred because of the python openssl library not able to load the certificates due to crypto error.

Resolution

1. Take powered down snapshots of all linked vCenters. 

2. Download Jxplorer and Java through the KB Using JXplorer to connect to the vSphere Single Sign-on

3. Navigate to Services > IdentityManager > Tenants > vsphere.local > Identity Providers > vsphere.local and remove both configurations (vmwSTSUpnSuffixes: SYSTEM-DOMAIN and vmwSTSAlias: SYSTEM-DOMAIN) as these are legacy configurations. 

4. Restart services on vCenter: service-control --stop --all && service-control --start --all

Additional Information

Powered by Wolken Software