vmdird-syslog.log in VCSA is flooded with error code 9207 and "Access deny on search"

Products

VMware vCenter Server

Issue/Introduction

/var/log/vmware/vmdird/vmdird-syslog.log in vCenter Server Appliance is being is flooded with the following error messages:

ERROR: SendSearchEntry failed DN=(cn=########-####-####-####-############,cn=ServiceRegistrations,cn=LookupService,cn=Country,cn=sites,cn=configuration,dc=vsphere,dc=local), (9207)()

WARNING: Access deny on search entry result [..\search.c,673] (bindedDN-cn=vcsa01.example.com,ou=Domain Controllers,dc=vsphere,dc=local) (targetDn-cn=########-####-####-####-############,cn=ServiceRegistrations,cn=LookupService,cn=Country,cn=sites,cn=configuration,dc=vsphere,dc=local)

In addition, when looking at /var/log/vmware/sso/vmware-sts-idmd.log, multipe "ldap error 50" messages can be seen as below:

[YYYY-MM-DDThh:mm:ss.####Z    ########-####-####-####-############ WARN ] [LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.WinLdapClientLibrary, error code: 50

[YYYY-MM-DDThh:mm:ss.####Z    ########-####-####-####-############ ERROR] [IdentityManager] Identity Manager failed to initialize

[YYYY-MM-DDThh:mm:ss.####Z    ########-####-####-####-############ ERROR] [ServerUtils] Exception 'com.vmware.identity.interop.ldap.InsufficientRightsLdapException: Insufficient Rights

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue occurs if the VCSA machine account was removed from the built in DCAdmins group. The machine account is used by the embedded platform services controller to access and manage the internal domain controller of the SSO domain, which stores its data in vmdird. Hence it requires this permission, which is normally given to it via membership in this internal SSO group

Resolution

Before applying the solution steps below, please ensure that a fresh backup or offline snapshot (in powered off state) of the vCenter Server was created. If the vCenter Server is in an Enhanced Linked Mode (ELM) replication with other vCenters, please be aware that offline snapshots must be created of all ELM nodes.

To solve this issue, add the VCSA machine account back to the DCAdmins group either by using Jxplorer or with an ldapadd command:

Option A - using Jxplorer

Connect to the embedded Platform Services Controller of the VCSA with the issue, using Jxplorer as described in Using JXplorer to connect to the vSphere Single Sign-on
Go in World > local > vsphere and find the vCenter Server machine account (named after the current vCenter PNID, usually the fully-qualified domain name of the vCenter). If a custom name other than vsphere.local was used for the embedded SSO domain, instead of local and vsphere, these will be the part of the custom domain name (again in reverse order)
right click this account and select "copy dn"
Now go in World > local > vsphere > Builtin and select DCAdmins
On the right-hand side, change to the "Table Editor" tab
In the table column attribute type, right-click on member and select Add another value
In the newly created row, under value, paste the DN copied in step 3
Click on the Submit button below the table, to save the change

Option B - using the ldapadd command

Connect to the vCenter Server Appliance via SSH
In the folder /tmp create a new text file called add_member.ldif
Populate the file with the following content (be aware that when using a different name for the embedded SSO domain other than vsphere.local, the dc values need to correspond to it)
```
dn: cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: cn=vcsa01.example.com,ou=Domain Controllers,dc=vsphere,dc=local
```
Run the following command to apply the information from the ldif file to the vmdir LDAP instance (replace <SSO-administrator_password> with the correct password):
```
#/opt/likewise/bin/ldapadd -h <psc fqdn> -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w '<SSO-administrator_password>' -f /tmp/add_member.ldif
```