Manually rebuild a missing group in VMDIR database
Article ID: 319409
Updated On:
Products
Issue/Introduction
The intention of this KB is to provide troubleshooting tools that allow engineers to rebuild groups in the VMDIR database. The example noted in the Symptoms section applies to the ActAsUsers group in VMDIR, which is only one of several. Use the methods in this KB to rebuild other groups as well if needed.
Symptoms:
Missing groups in VMDIR can cause authentication failures in different vCenter clients, leading to login failures. In this KB we will discuss a missing ActAsUsers group in VMDIR and the associated symptoms, other groups might lead to differing symptoms.
If the ActAsUsers group is missing, you will see the following symptoms:
Log Messages:
/var/log/vmware/vsphere_client/logs/vsphere_client_virgo.log
/var/log/vmware/vsphere_ui/logs/vsphere_client_virgo.log
[YYYY-MM-DDTHH:MM:SS] [INFO ] http-bio-9090-exec-9 70001686 100021 ###### com.vmware.vsphere.client.security.websso.MetadataGeneratorImpl SP url generated is https://<FQDN>/vsphere-client
[YYYY-MM-DDTHH:MM:SS] [ERROR] http-bio-9090-exec-9 70001686 100021 ###### com.vmware.vsphere.client.security.websso.MetadataGeneratorImpl Error when registering ngc metadata. com.vmware.vim.binding.sso.fault.NoPermission: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:383)
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)
/var/log/vmware/sso/ssoAdminserver.log
[YYYY-MM-DDTHH:MM:SS pool-2-thread-3 opId=ac69024a-d163-4810-a1a7-400a1b6e4ab5 INFO com.vmware.identity.vlsi.RoleBasedAuthorizer] User {Name: vsphere-webclient-<machine-id>, Domain: vsphere.local} with role 'GuestUser' is authorized for method call 'ConfigurationManagementService.getIssuersCertificates'
[YYYY-MM-DDTHH:MM:SS pool-2-thread-5 opId=fbfb6980-80ff-4373-8e17-4b97c991bcfa INFO com.vmware.identity.vlsi.RoleBasedAuthorizer] User {Name: vsphere-webclient-<machine-id>, Domain: vsphere.local} with role 'GuestUser' is authorized for method call 'SessionManager.login'
[YYYY-MM-DDTHH:MM:SS pool-2-thread-2 opId=9f5cd667-c2a4-4248-915a-23f7d9189760 INFO com.vmware.identity.vlsi.RoleBasedAuthorizer] User {Name: vsphere-webclient-<machine-id>, Domain: vsphere.local} with role 'GuestUser' is authorized for method call 'ServiceInstance.retrieveServiceContent'
[YYYY-MM-DDTHH:MM:SS pool-2-thread-5 opId=8a8dcf2a-94fa-4d75-a751-363a582b405c INFO com.vmware.identity.vlsi.RoleBasedAuthorizer] User {Name: vsphere-webclient-<machine-id>, Domain: vsphere.local} with role 'GuestUser' is notauthorized for method call 'ConfigurationManagementService.importSAMLMetadata'
When logging into the Web Client for the vCenter, you will be presented with the following message:
vSphere 6.0 Error:
Web client: "A server error occurred. No error message is available. :-("
vSphere 6.5 Error:
Web client and HTML5 client: "The request is invalid. Reason:ns0:InvalidRequest: Access not authorized!"
vSphere 6.7 Error:
- HTML5 client: "Cannot connect to vCenter Single Sign-On server:https://vcenterfqdn/sts/STSService/vsphere.local."
- Flash Client: "The request is invalid. Reason:nso:InvalidRequest: Access not authorized!"
Log into JXplorer to view the VMDIR database: Using JXplorer to connect to the vSphere Single Sign-on
ActAsUsers Group should show up like this:
Here is an image of the missing ActAsUsers group in VMDIR:
Environment
VMware vCenter Sever Appliance 6.x
Cause
The ActAsUsers group can be deleted manually from the vCenter web client global permissions view by customers. The group doesn't list any users in the web client despite the fact that vpxd-extension and vsphere-webclient are actually members of this group in the VMDIR database. This might lead customers to think it is safe to delete.
Resolution
In order to correct the missing group in AD, gather the following details:
- Solution User ID's for vsphere-webclient and vpxd-extension.
- SSO administrator credentials.
- LDIF export from the same version PSC. *** Analysis on this from 6.0 and 6.7 labs show that they're the same for the ActAsUsers Group, so for this use case, we should be fine with the detail I have noted below.
- MachineID of the vCenter this group is missing from.
To gather these details:
1. Gather the SolutionUser ID's using the following command run against the PSC (use SSO Administrator credentials when prompted):
# /usr/lib/vmware-vmafd/bin/dir-cli service list
2. SSO Administrator credentials must be provided by the customer.
3. Extract LDIF export for the ActAsUsers group from same version PSC. (in case we need to do this on some other VMDIR CN object): How to export VMDir information from vCenter Server Appliance
# /opt/likewise/bin/ldapsearch -b "cn=ActAsUsers,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO Admin Password'
Command output will look like:
# extended LDIF
#
# LDAPv3
# base <cn=ActAsUsers,dc=vsphere,dc=local> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#
# ActAsUsers, vsphere.local
dn: CN=ActAsUsers,DC=vsphere,DC=local
member: CN=vsphere-webclient-########-####-####-####-###########,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=vpxd-extension-########-####-####-####-###########,CN=ServicePrincipals,DC=vsphere,DC=local
nTSecurityDescriptor:: <SSL info here>
groupType: 2
description: Act-As Users
sAMAccountName: ActAsUsers
cn: ActAsUsers
objectClass: top
objectClass: group
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
- nTSecurityDescriptor can be removed, this will be added back when we import the entry.
- The Members in this output are unique to the vCenter from which they are exported.
- The DC entries will also be unique to the vSphere SSO domain alias. For @vsphere.local we will have DC=vsphere,DC=local. For @sso.local, we will have DC=sso,DC=local
4. Gather the Machine ID with the following command, run from the vCenter:
# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost
=====================================================================================
Here are the steps to Import the Group after identifying the required details:
Command to import the ActAsUsers group back to VMDIR: (To be run on the PSC), Note: copy the entire entry from <<EOF to EOF
# /opt/likewise/bin/ldapadd -h localhost -x -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSO Admin password' <<EOF
dn: CN=ActAsUsers,DC=vsphere,DC=local
member: CN=vsphere-webclient-########-####-####-####-###########,CN=ServicePrincipals,DC=vsphere,DC=local
member: CN=vpxd-extension-########-####-####-####-###########,CN=ServicePrincipals,DC=vsphere,DC=local
groupType: 2
description: Act-As Users
sAMAccountName: ActAsUsers
cn: ActAsUsers
objectClass: top
objectClass: group
EOF
Note: Modify the number following member name vsphere-webclient and vpxd-extension in this command using the machineID of the vCenter.
After you run this command, the Group should show up in jXplorer again and users should be able to log into the web client again.
The LDAPADD command will execute successfully with the following message:
adding new entry "CN=ActAsUsers,DC=vsphere,DC=local"
It will return a failed message if you use the wrong machineID and will fail to import. Here's the message you'll see with an incorrect MachineID:
adding new entry "CN=ActAsUsers,DC=vsphere,DC=local"
ldap_add: Constraint violation (19)
additional info: BEChkDNRef (9705)((Unknown error 9703)(CN=vsphere-webclient-########-####-####-####-###########
Workaround:
An alternate method for adding this group back via SSH on the PSC is to use the dir-cli command. Here are example commands for this method:
1. Create the SSO group:
# /usr/lib/vmware-vmafd/bin/dir-cli ssogroup create --name ActAsUsers
Result:
Service [ActAsUsers] created successfully
2. Add both the vsphere-webclient and vpxd-extension users to the new group:
# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name ActAsUsers --add vpxd-extension-########-####-####-####-###########
Result:
Account [vpxd-extension-########-####-####-####-###########] added to group [ActAsUsers]
# /usr/lib/vmware-vmafd/bin/dir-cli group modify --name ActAsUsers --add vsphere-webclient-########-####-####-####-###########
Result:
Account [vsphere-webclient-########-####-####-####-###########] added to group [ActAsUsers]
Additional Information
Identify partner PSC's replication: Determining replication agreements and status with the Platform Services Controller (PSC)
KB identifying how to capture an LDIF export: How to export VMDir information from vCenter Server Appliance
Impact/Risks:
Editing the VMDIR database is a HIGH RISK operation. Prior to applying any of the changes noted in this KB, you MUST have the following:
1. A complete understanding of the PSC and vCenter architecture, Including all linked PSC's.
2. Powered off snapshots of ALL PSC and vCenter nodes in the environment.
3. JXplorer connectivity to PSCs.
If unsure of the steps in this process, open an SR to review prior to applying any changes.
Feedback
