The vCenter Server is inaccessible because the vpxd-svcs service failed to initiate

Products

• VMware SDDC Manager
• VMware vCenter Server 8.0

Issue/Introduction

vpxd-svcs service failed to start with following errors from vpxd-svcs.log logs.

• /var/log/vmware/vpxd-svcs/vpxd-svcs.log

<YYYY-MM-DD>T<time> [Thread-11 [] ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP fault
 com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.

 

• /var/log/vmware/trustmanagement/trustmanagement-svcs.log

<YYYY-MM-DD>T<time> [inventoryPermissionConverterScheduler-1 [] ERROR com.vmware.vcenter.trustmanagement.migration.InventoryPermissionConverter  opId=] VPXD AuthZ inventory permission conversion failed
com.vmware.svcaccount.token.exceptions.AcquireTokenException: SAML token request was rejected
<snip> 
Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

 

• /var/log/vmware/applmgmt/applmgmt.log

<YYYY-MM-DD>T<time> [3179]ERROR:vmware.appliance.extensions.authentication.authentication_sso:Unhandled exception during SAML token validation
Traceback (most recent call last):
<snip>
  File "/usr/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1825, in load_certificate
    _raise_current_error()
<snip>
OpenSSL.crypto.Error: []
<YYYY-MM-DD>T<time> [3179]ERROR:vmware.appliance.vapi.auth:Could not parse HOK Token
Traceback (most recent call last):
 <snip>
  File "/usr/lib/python3.7/site-packages/OpenSSL/crypto.py", line 1825, in load_certificate
    _raise_current_error()
<snip>
OpenSSL.crypto.Error: []

 

ldif collected from vCenter looks like this: 

dn: cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local
objectClass: vmwSTSIdentityStore
objectClass: top
cn: vsphere.local
vmwSTSAlias: SYSTEM-DOMAIN  <------ Legacy Configuration 
vmwSTSAuthenticationType: SRP
vmwSTSConnectionStrings:  ldap://<vCenter Server FQDN>:389 
vmwSTSDomainName: vsphere.local
vmwSTSDomainType: SYSTEM_DOMAIN
vmwSTSGroupBaseDN: DC=vsphere,DC=local
vmwSTSProviderType: IDENTITY_STORE_TYPE_VMWARE_DIRECTORY
vmwSTSServiceUseMachineAccount: false
vmwSTSTimeout: 0
vmwSTSUpnSuffixes: SYSTEM-DOMAIN  <------Legacy Configuration
vmwSTSUserBaseDN: DC=vsphere,DC=local

Environment

VMware vCenter Server 8.0

Cause

Issue occurred because of the python openssl library not able to load the certificates due to crypto error.

Resolution

1. Take powered down snapshots of all linked vCenters. 

2. Download Jxplorer and Java through the KB Using JXplorer to connect to the vSphere Single Sign-on

3. Navigate to Services > IdentityManager > Tenants > vsphere.local > Identity Providers > vsphere.local and remove both configurations (vmwSTSUpnSuffixes: SYSTEM-DOMAIN and vmwSTSAlias: SYSTEM-DOMAIN) as these are legacy configurations. 

4. Restart services on vCenter: service-control --stop --all && service-control --start --all

Additional Information

How to export VMDir information from vCenter Server Appliance