Unable to remove stale entries from jxplorer due to vmdird database corruption

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

Unable to remove replication partner or servers using jxplorer / cmsso unregister command (Decommission VC / PSC)

Following error in /var/log/vmware/vmdird/vmdird-syslog.log:

javax.naming.NameNotFoundException: [LDAP: error code 32 - DNToEID (9703)((MDB_NOTFOUND: No matching key/data pair found)(labeleduri=ldap://####,cn=replication agreements,cn=####,cn=servers,cn=default-site,cn=sites,cn=configuration,dc=vsphere,dc=local))]; remaining name '"labeledURI=ldap://####,cn=Replication Agreements,cn=####,cn=Servers,cn=Default-site,cn=Sites,cn=Configuration,dc=vsphere,dc=local"

Removing the servers from jxplorer gives the following error:
Unable to perform Modify operation.

Environment

vCenter server 7.x
vCenter server 8.x

Cause

This issue occurs when the vmdir database contains stale, orphaned, or corrupted entries.

Resolution

Prerequisites (CRITICAL):

Before following the below resolution, the following steps must be completed to prevent data loss or service degradation:

Log into the vCenter Server Appliance Management Interface and execute a successful file-based backup.
Power off the vCenter Server VM and take a cold (offline) snapshot by logging directly into the ESXi host managing the VC VM.
For environments using Enhanced Linked Mode (ELM), ensure simultaneous offline snapshots are taken for all linked vCenter nodes before proceeding.
Confirm that the vCenter Server has valid, resolvable forward (A) and reverse (PTR) DNS entries configured for the SSO domain.

The primary method to unregister or decommission stale servers from vCenter is by running the cmsso-util unregister command as follows:

cmsso-util unregister --node-pnid External_PSC --username administrator@your_domain_name --passwd 'vCenter_Single_Sign_On_password'

If the vmDIR database is corrupted, the cmsso-util unregister command will fail with an error similar to the following:

Solution users, computer account and service endpoints will be unregistered
Could not find a host id which maps to ####. Solution users will not be removed.
YYYY-MM-DDTHH:MM:SS  RC = 32
Stdout = vdcleavefd offline for server ####
Leave federation cleanup failed. Error[32] - No such object

If this error is received, the only way to clear the stale entry and resolve the database corruption is to perform an intra-domain repoint (repointing the vCenter to itself).

**CAUTION: DATA LOSS PREVENTION**

Executing an intra-domain repoint completely rebuilds/wipes the Single Sign-On (SSO) database.The following configurations will be lost and must be manually documented and recreated post-repoint:

Identity Sources: Go to Administration > Single Sign On > Identity Sources. Capture screenshots of all server URIs, Base DNs, Bind DNs, and Domain Aliases. Download any LDAPS certificates if applicable.
Global Permissions: Go to Administration > Global Permissions. Document all AD groups/users assigned at the global level and their associated roles.
Custom Roles: Go to Administration > Roles. Document the names and exact privilege mappings of all custom roles. (Alternatively, use PowerCLI to export Custom Roles and Global Permissions to JSON).
Local SSO Users: Document any users created directly within the vsphere.local domain (excluding the default administrator).
Record Current LDU-GUID: Capture and safely store the current LDU-GUID before proceeding by running this command via SSH:
/usr/lib/vmware-vmafd/bin/vmafd-cli get-ldu --server-name localhost

To fix the database corruption, perform an intra-domain repoint to itself:

cmsso-util domain-repoint -m execute --src-emb-admin Administrator  --dest-domain-name vsphere.local