vpxd-svcs failing to start with "Could not find any solution users from SSO" due to machine account missing from certain groups

Products

• VMware vCenter Server

Issue/Introduction

vCenters in ELM

vpxd-svcs failing to start with:

ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceCommunicationException while fetching admin group for the SSO Admin user : [email protected]
com.vmware.cis.server.ssoauthentication.exception.ServiceCommunicationException: com.vmware.vim.sso.admin.exception.NoPermissionException

and

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterFactory]: Constructor threw exception; nested exception is java.lang.RuntimeException: Could not find any solution users from SSO.

 

Replication was previously changes behind due to machine account password mismatches - fixed with dir-cli method from KB https://broadcomcms-software.wolkenservicedesk.com/external/article?articleNumber=319348

PNID changes previously took place in the environment - specifically for changing the VCs PNID case (uppercase to lowercase or the other way round) - one or more of these PNID changes initially failed and only a single VC was reverted to snapshot during it, causing replication to be changes behind

 

Environment

VC 8.0

Cause

During the PNID change operation, the old machine account is removed from the SSO database and then the new machine account with the new VC PNID is added to the SSO database as part of the normal process

If the PNID change fails and only a single VC is reverted to snapshot in the SSO domain, replication becomes broken in most cases

After replication is fixed, the VC which wasn't reverted to snapshot will send the update to the issue VC to remove references for the machine account of the issue VC from certain groups - causing the issue

Resolution

Ensure offline snapshots of all nodes in the SSO domain are taken before commencing.

The below assumes the SSO domain is the default vsphere.local - adjust this as needed for custom SSO names

 

Group memberships can be checked with dir-cli:

/usr/lib/vmware-vmafd/bin/dir-cli group list --name <GROUP_NAME>

 

Readd the machine account to the following groups if missing:

Administrators

DCAdmins

 

To readd the machine account to the group use the below command. Run the below command by pasting each line individually and pressing enter. After EOF the user will be prompted for the SSO admin password:

/opt/likewise/bin/ldapmodify -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
dn: cn=<GROUP_NAME>,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: cn=<VC_FQDN>,ou=Domain Controllers,DCvsphere,DC=local
EOF

 

To confirm it was successful, check the group with dir-cli:

/usr/lib/vmware-vmafd/bin/dir-cli group list --name <GROUP_NAME>

 

Restart services on the vCenter to apply:

service-control --stop --all && service-control --start --all

Additional Information

Additional solution users may be missing from their group memberships - to check and fix, use the script from KB Fixing missing SSO Group Memberships for vSphere Solution Users with the solution_users_fixer script