Removing stale lookupservice service registrations from vmdir using ldapdelete on vCenter Server

Products

VMware vCenter Server

Issue/Introduction

Stale service registrations persist in the vCenter Server lookupservice after the associated appliances (such as vSphere Replication or Site Recovery Manager) have been decommissioned.
Running lstool.py unregister appears to succeed or returns no error, but the entries reappear or remain visible in lstool.py list output.
In Enhanced Linked Mode (ELM) environments, stale entries removed from one vCenter node replicate back from the partner node through vmdir replication.

Environment

vCenter Server 7.x or 8.x
Environments using Enhanced Linked Mode (ELM) with two or more linked vCenter nodes

Cause

The lstool.py unregister command operates at the lookupservice SDK layer. In some cases, it updates the lookupservice's in-memory state without fully removing the underlying LDAP objects from the vmdir database. Each lookupservice service registration is stored in vmdir as a parent object with child entries (endpoints and properties). If these child objects are not removed, the parent entry cannot be fully deleted from the directory, and the stale registration returns when the lookupservice refreshes its cache or restarts.

In ELM environments, vmdir replication between partner nodes can also restore entries that were only partially removed from one side.

Resolution

Before you begin:

Take powered-off snapshots of all linked vCenter VMs in order to secure a safe rollback point
Identify the stale entries. You can verify which service registrations are stale by running lstool.py list and checking for entries that reference decommissioned IP addresses or hostnames.

Note: Placeholder text is indicated by the <> symbols. Replace any of them in the commands with the actual data required.

Step 1: Export the vmdir database

SSH to the vCenter Server Appliance.

Run the following command to export the full vmdir contents to an LDIF file:

/opt/likewise/bin/ldapsearch -b "dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W > /tmp/vmdir_export.ldif

Enter the administrator@<domain>.local password when prompted.
Repeat on each linked vCenter node in the ELM domain.

Step 2: Identify the stale LDAP entries

Search the exported LDIF file for the decommissioned IP addresses or hostnames:
```
grep -n "<decommissioned-ip-or-hostname>" /tmp/vmdir_export.ldif
```
The matching lines show endpoint URLs (vmwLKUPURI) within the lookupservice registration tree. Note the line numbers.

Read the surrounding lines to find the parent service registration DN. The structure follows this pattern:

dn: cn=<service-id>,cn=ServiceRegistrations,cn=LookupService,cn=default-site, cn=Sites,cn=Configuration,dc=vsphere,dc=local

For each stale parent entry, identify its child objects. Each service registration typically has three children:

cn=Endpoint1,cn=<service-id>,cn=ServiceRegistrations,...
cn=Endpoint2,cn=<service-id>,cn=ServiceRegistrations,...
cn=Property1,cn=<service-id>,cn=ServiceRegistrations,...

Record the full DN of each child and parent entry. Preserve the exact spacing in the DN string as it appears in the LDIF. For example, there may be a space after cn=default-site, in the DN.

Step 3: Delete the stale entries from vmdir

LDAP directory entries with children cannot be deleted until the children are removed first. Delete in this order: children first, then parents.

For each stale service registration, delete the child entries first:

/opt/likewise/bin/ldapdelete -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W "<full-DN-of-Endpoint1>"
/opt/likewise/bin/ldapdelete -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W "<full-DN-of-Endpoint2>"
/opt/likewise/bin/ldapdelete -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W "<full-DN-of-Property1>"

After all children are removed, delete the parent entry.:

/opt/likewise/bin/ldapdelete -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W "<full-DN-of-service-registration>"

Enter the [email protected] password when prompted for each command.
In ELM environments, run the same delete commands on all linked vCenter nodes as close to the same time as possible. This prevents vmdir replication from restoring the entries from a partner node before that node is also cleaned.

Step 4: Verify the cleanup

Confirm the LDAP objects are gone by searching the ServiceRegistrations container:

/opt/likewise/bin/ldapsearch -b "cn=ServiceRegistrations,cn=LookupService,cn=default-site, cn=Sites,cn=Configuration,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W "(cn=<service-id>*)"

This should return no results for the deleted entries.

Verify the lookupservice view is clean:

python /usr/lib/vmware-lookupsvc/tools/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert | grep -i "<decommissioned-ip-or-hostname>"

This should return no results.

If lstool.py list still shows the deleted entries after the LDAP objects are confirmed gone, restart the lookupservice to clear its in-memory cache:
```
vmon-cli --restart lookupsvc
```
Wait at least two minutes, then re-run the lstool.py list check.

Additional Information

For the standard lookupservice cleanup procedure using lstool.py unregister, see Script to remove stale service registrations from vCenter Server.
If the stale entries also have associated solution user accounts in vmdir, those can be removed using dir-cli service delete. In many cases involving decommissioned appliances, no solution user accounts exist.
In ELM environments, vmdir replicates changes between partner nodes approximately every 30 seconds. When performing cleanup, minimize the time between deleting entries on the first node and deleting them on subsequent nodes.