service-control --status :Stopped: vmware-certificatemanagement vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vpxd-svcs vsphere-ui vstatsReviewing "vpxd-svcs" service log shows errors such as:
/var/log/vmware/vpxd-svcs/vpxd-svcs.log
YYYY-MM-DDTHH:MM:SS.###Z [Thread-13 [] INFO com.vmware.cis.server.util.impl.InitPoolTask opId=] Initializing VC connection poolYYYY-MM-DDTHH:MM:SS.###Z [main [] ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] SOAP faultcom.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure. at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
Reviewing SSO service log shows errors such as:
/var/log/vmware/sso/vmware-identity-sts.log
YYYY-MM-DDTHH:MM:SS.###Z INFO sts[86:tomcat-http--40] [CorId=######-####-#######] [com.vmware.identity.idm.server.IdentityManager] User vpxd-svcs-user-########-####-####-####-###########@vsphere.local attempting to login via unsupported domain provider vsphere.local type com.vmware.identity.idm.server.provider.vmwdirectory.SystemDomainAliasedProvider on federated tenant vsphere.local. This is not supported.YYYY-MM-DDTHH:MM:SS.###Z ERROR sts[86:tomcat-http--40] [CorId=######-####-#######]] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [vpxd-svcs-user-########-####-####-####-###########@vsphere.local] for tenant [vsphere.local]com.vmware.identity.idm.IDMLoginException: Invalid user principal 'vpxd-svcs-user-########-####-####-####-###########@vsphere.local'. at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3159)Caused by: com.vmware.identity.sts.idm.InvalidCredentialsException: com.vmware.identity.idm.IDMLoginException: Invalid user principal 'vpxd-svcs-user-########-####-####-####-###########@vsphere.local'.
This issue is caused by the persistence of a legacy identity source SYSTEM-DOMAIN, following a historical migration from vCenter Server 5.1 to version 5.5 or later.
Remove the SYSTEM-DOMAIN legacy domain entries by following the instructions in the KB Fix legacy SYSTEM-DOMAIN artifacts in vCenter Servers upgraded from vCenter Server 5.1+.