vpxd-svcs fails to start | Provided credentials are not valid.| ERROR_NO_SUCH_GROUP (1319)

Products

• VMware vCenter Server

Issue/Introduction

/var/log/vmware/vpxd-svcs/vpxd-svcs.log

YYYY-MM-DDTHH:MM:SS [Thread-14 [] INFO com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor  opId=] Provided credentials are not valid.
YYYY-MM-DDTHH:MM:SS [Thread-14 [] WARN  com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider  opId=] Refreshing STS client due to authentication failure
com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenServiceImpl.java:1066) ~[libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:988) ~[libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902) ~[libwstclient.jar:?]
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509) ~[libwstclient.jar:?]
        at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.acquireSamlToken(SolutionTokenProvider.java:54) [inventory-server.jar:?]
        at com.vmware.cis.server.ssoauthentication.impl.AbstractTokenProvider.getSamlToken(AbstractTokenProvider.java:42) [inventory-server.jar:?]
        at com.vmware.cis.server.util.VpxdClient.loginBySamlToken(VpxdClient.java:181) [inventory-server.jar:?]
        at com.vmware.cis.server.util.VpxdClient.login(VpxdClient.java:78) [inventory-server.jar:?]
        at com.vmware.cis.server.util.ConnectionManager$1.makeObject(ConnectionManager.java:159) [inventory-server.jar:?]
        at com.vmware.cis.server.util.ConnectionManager$1.makeObject(ConnectionManager.java:149) [inventory-server.jar:?]
        at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1691) [commons-pool-1.6.jar:1.6]
        at com.vmware.cis.server.util.impl.InitPoolTask.run(InitPoolTask.java:44) [inventory-server.jar:?]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362]
YYYY-MM-DDTHH:MM:SS [Thread-14 [] INFO  com.vmware.cis.server.ssoauthentication.impl.ServiceLocatorImpl  opId=] Security Token Service URL is http://localhost:1080/sts/system-STSService
YYYY-MM-DDTHH:MM:SS [Thread-14 [] INFO  com.vmware.cis.server.ssoauthentication.impl.ServiceLocatorImpl  opId=] SSO Admin URL is http://localhost:1080/sso-adminserver/system-sdk
YYYY-MM-DDTHH:MM:SS [Thread-14 [] INFO  com.vmware.cis.server.ssoauthentication.impl.ServiceLocatorImpl  opId=] SSO GroupCheckEndPoint URL is http://localhost:1080/sso-adminserver/system-sdk
YYYY-MM-DDTHH:MM:SS [Thread-14 [] ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.

Resetting solution user certificate using /usr/lib/vmware-vmca/bin/certificate-manager fails. 
 

Running lsdoctor -u fails with the below error :

YYYY-MM-DDTHH:MM:SS  RC = 39
Stdout =
Stderr = dir-cli failed. Error 1319: Operation failed with error ERROR_NO_SUCH_GROUP (1319)
YYYY-MM-DDTHH:MM:SS INFO checkAndFix: Execution finished - Please check log for details

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

This issue happens because SolutionUsers group is missing. 

Resolution

To verify the existence of the Solution Users group, execute the following LDAP search command, ensuring to replace "vsphere.local" with the actual SSO domain used in the customer environment:

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "dc=vsphere,dc=local" -s sub "sAMAccountName=SolutionUsers" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD'

We can create the missing SolutionUsers group using the dir-cli command.

/usr/lib/vmware-vmafd/bin/dir-cli ssogroup create --name SolutionUsers --description "Well-known solution users' group, which contains all solution users as members."

After creating the SolutionUsers group with the dir-cli command, we can confirm its creation using the ldapsearch command. Once verified, we can proceed with the solution user certificate replacement.

lsdoctor -u