Unable to remove an Identity Source from vCenter server: "Unexpected error encountered while removing selected identity source"

Products

• VMware vCenter Server

Issue/Introduction

• Removal of an identity provider from the vCenter UI fails with an "Unexpected error encountered while removing selected identity source" error.
  
  
  
• The identity source cannot be deleted through the command line using the sso-config.sh script or direct ldapdelete command.
  

  sso-config.sh -delete_identity_source -i identity_source_name
ERROR: Operations error
com.vmware.identity.idm.IDMException: Operations error
        at com.vmware.identity.idm.server.ServerUtils.getRemoteException(ServerUtils.java:174)
        at com.vmware.identity.idm.server.IdentityManager.deleteProvider(IdentityManager.java:10243)
        at com.vmware.identity.idm.client.CasIdmClient.deleteProvider(CasIdmClient.java:870)
        at com.vmware.identity.ssoconfig.SsoConfig.deleteIdentitySources(SsoConfig.java:2312)
        at com.vmware.identity.ssoconfig.SsoConfig.main(SsoConfig.java:503)
Caused by: com.vmware.identity.interop.ldap.OperationsErrorLdapException: Operations error
LDAP error [code: 1]

  
  /opt/likewise/bin/ldapdelete -r "cn=identity_source-name,cn=IdentityProviders,cn=domain.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=domain,dc=local" -D "cn=Administrator,cn=Users,dc=domain,dc=local" -W
ldap_delete: Operations error (1)
ldap_prune: Operations error (1)
ldap_delete: Operation not allowed on non-leaf (66)
additional info: Delete of a non-leaf node is not allowed.

• The /var/log/vmware/vmdird/vmdird.log log on the vCenter Server contains the following error entries:
  YYYY-MM-DD:THH:MM:SS:t@#######:ERROR: MdbValidateAttrUniqueness detected that attr 'entryDN' value 'cn=attributemapping-0#objectguid:########-####-####-####-############,cn=deleted objects,dc=domain,dc=local' already exists in scope 'cn=DSE Root', will return error 9102
YYYY-MM-DD:THH:MM:SS:t@#######:ERROR: InternalDeleteEntry: VdirExecutePostDeleteCommitPlugins - code(9700)
YYYY-MM-DD:THH:MM:SS:t@#######:ERROR: VmDirSendLdapResult: Request (Delete), Error (LDAP_OPERATIONS_ERROR(1)), Message (BEEntryDelete (9700)((Unknown error 9102)(entryDN))), (0) socket (127.0.0.1)
YYYY-MM-DD:THH:MM:SS:t@#######:ERROR: InternalDeleteEntry: DN= cn=identity_source_name,cn=IdentityProviders,cn=domain.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=domain,dc=local is a non-leaf node
YYYY-MM-DD:THH:MM:SS:t@#######:ERROR: InternalDeleteEntry: VdirExecutePostDeleteCommitPlugins - code(9117)
YYYY-MM-DD:THH:MM:SS:t@#######:ERROR: VmDirSendLdapResult: Request (Delete), Error (LDAP_NOT_ALLOWED_ON_NONLEAF(66)), Message (Delete of a non-leaf node is not allowed.), (0) socket (127.0.0.1)
  
  
• Issue is observed only when multiple vCenter servers are in an Enhanced Linked Mode

Environment

VMware vCenter Server 8.x

Cause

This issue stems from concurrent identity source additions and removals across multiple nodes within an Enhanced Linked Mode (ELM) environment.
VMDIR does not support scenarios where a child object is created under a parent on one node while that same parent object is deleted on another node during the same replication cycle.

Resolution

The tombstone entries for the parent node must be manually purged from VMDIR. 
Since the default tombstone cleanup cycle only targets objects older than 45 days once every 24 hours, this process must be accelerated to address the stale issue. 
By temporarily reducing the required object age threshold, you can force the immediate purging of newer tombstones and clear the replication conflict.

Note: Tombstone removal does not replicate between vCenters in a SSO domain and must be performed on all the participating nodes.

1. Verify the existing registry values in lwregshell by executing the following command:
   /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmdir\Parameters]"
   
   
2. Execute the following command to reduce the TombstoneReapingThreadFreqInSec from its default 24-hour interval. Setting this to 3600 seconds (1 hour) is the lowest supported value and will increase the frequency of the tombstone cleanup process:
   /opt/likewise/bin/lwregshell add_value "[HKEY_THIS_MACHINE\\Services\\vmdir\\Parameters]" TombstoneReapingThreadFreqInSec REG_DWORD 3600
   
   If the TombstoneReapingThreadFreqInSec value already exists in the Registry, modify it using the set_value command to update the frequency to 3600 seconds.
   /opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\\Services\\vmdir\\Parameters]" TombstoneReapingThreadFreqInSec 3600
   
   
3. To modify the default 45-day object retention period, create the TombstoneExpirationPeriodInSec registry key and set it to 3600 seconds (1 hour):
   /opt/likewise/bin/lwregshell add_value "[HKEY_THIS_MACHINE\\Services\\vmdir\\Parameters]" TombstoneExpirationPeriodInSec REG_DWORD 3600
   
   If the TombstoneExpirationPeriodInSec value is already present in the registry, use the set_value command to update the expiration threshold to 3600 seconds (1 hour):
   /opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\\Services\\vmdir\\Parameters]" TombstoneExpirationPeriodInSec 3600
   
   
4. Restart the VMware Directory Service (VMDIR) to apply the registry configuration changes:
   /opt/likewise/bin/lwsm restart vmdir
   
   
5. Monitor the tombstone cleanup progress by reviewing the /var/log/vmware/vmdird/vmdird.log file. Successful execution is indicated by the following log signatures:
   YYYY-MM-DD:THH:MM:SS:t@#######:INFO: Tombstone reaping starting. Expiry period (3600), Reap Hz (3600), Batch count (107)
YYYY-MM-DD:THH:MM:SS:t@#######:INFO: Tombstone reaping ended
6. Once the tombstone purge has successfully completed, proceed with the removal of the affected identity source from the vCenter configuration.