Unable to remove stale entries from jxplorer due to vmdird database corruption

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• Unable to remove replication partner or servers using jxplorer / cmsso unregister command (Decommission VC / PSC)
• Following error in vmdird-syslog.log:

/var/log/vmware/vmdird/vmdird-syslog.log

javax.naming.NameNotFoundException: [LDAP: error code 32 - DNToEID (9703)((MDB_NOTFOUND: No matching key/data pair found)(labeleduri=ldap://xxx,cn=replication agreements,cn=xxxxx,cn=servers,cn=default-site,cn=sites,cn=configuration,dc=vsphere,dc=local))]; remaining name '"labeledURI=ldap://xxx,cn=Replication Agreements,cn=xxxx,cn=Servers,cn=Default-site,cn=Sites,cn=Configuration,dc=vsphere,dc=local"

• Removing the servers from jxplorer gives the following error:

Environment

vCenter server 7.x
vCenter server 8.x

Cause

VMDIRD database is corrupted

Resolution

This is a destructive task and would require proper snapshot of vCenter server and PSC if involved.

NOTE: If it is a standalone VC, take an online snapshot
If it is in linked mode take powered off snapshots of all nodes.

1. Primary command to unregister or decommission stale servers from vCenter server is by running cmsso unregister command as follows:

cmsso-util unregister --node-pnid External_PSC --username administrator@your_domain_name --passwd 'vCenter_Single_Sign_On_password'

2. In case if cmsso unregister command gives the error as below, only way to remove the stale entry is to perform a domain repoint to itself 

Solution users, computer account and service endpoints will be unregistered
Could not find a host id which maps to xxxxx. Solution users will not be removed.
2024-09-01T10:45:30.667Z  RC = 32
Stdout = vdcleavefd offline for server xxxxxxxx
Leave federation cleanup failed. Error[32] - No such object

3. To fix the Database corruption perform a domain repoint to itself (Take a backup of all domain users from global permission and identity source configuration)

cmsso-util domain-repoint -m execute --src-emb-admin Administrator  --dest-domain-name vsphere.local