The VMware Directory Service (VMDIRD) is entering an unrecoverable state with error "vdirReplicationThrFun: Replication has failed with unrecoverable error".
search cancel

The VMware Directory Service (VMDIRD) is entering an unrecoverable state with error "vdirReplicationThrFun: Replication has failed with unrecoverable error".

book

Article ID: 430325

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Addition, deletion or modification of the same object represented by the same DN(Distinguished Name) in VMDir within the same replication cycle, can lead to replication issues ultimately resulting the VMDir in failure/unrecoverable state. Operations leading to such behaviour have to be avoided.  These issues typically occur when a change made on the same DN ldap object on one vCenter Server is repeated on a partner vCenter before the initial replication cycle completes across the environment.   . VMDir replication cycle is a duration of 30 seconds. When a change happens in VMDir(add/modify/delete), it gets propagated to its partners’ in the next pull request of the consecutive replication cycle.Now when even before this change is propagated, if some ldap operation is carried out on the same DN ldap object , it results in replication conflicts especially when the ldap entry is a part of hierarchal data in VMDir. Such replication conflicts get piled up over time, which lead VMDir to a state wherein it cannot pull anymore replication changes from it’s partners and getting into failure state.

 

Example log snippets

 /var/log/vmware/vmdird/vmdird.log

 

2026-02-08T05:05:02.521Z WARNING vmdir ########### [vc@4413] ReplAddEntryVmDirInternalAddEntry: 32 (Parent object does not exist). DN: cn=urn:vmomi:InventoryServiceTag:################################:GLOBAL,cn=urn:vmomi:InventoryServiceCategory:#########################:GLOBAL,cn=urn:vmomi:InventoryServiceScope:default-scope:GLOBAL,cn=Tagging,cn=services,dc=vsphere,dc=local, first attribute: cn, it's meta data: '############################' NOT resolving this possible replication CONFLICT or out-of-parent-child-order replication scenario. For this subtree, system may not converge. Partner USN #####

 

2026-02-08T05:05:02.579Z ERROR vmdir ############### [vc@####] _VmDirProcessReplicationPage: sync_state = (1) dn = (cn=AssetEntity_host-##################,cn=LicenseService,cn=services,dc=vsphere,dc=local) error = (68)

2026-02-08T05:05:02.579Z ERROR vmdir ############ [vc@####] _VmDirConsumePartner: Did not succesfully perform any updates after full pull. Moving vmdir to an unrecoverable state

2026-02-08T05:05:02.581Z INFO vmdir ############ [vc@####] VmDir State (5)

2026-02-08T05:05:02.581Z INFO vmdir ############ [vc@####] VmDir read-only reason (0)

2026-02-08T05:05:02.581Z ERROR vmdir ############# [vc@####] vdirReplicationThrFun: Replication has failed with unrecoverable error.

2026-02-08T05:05:02.754Z ERROR vmdir ############# [vc@####] _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.

2026-02-08T05:05:02.754Z ERROR vmdir ############# [vc@####] VmDirSendLdapResult: Request (Search), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (######)

 

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

Cause

Addition, deletion or modification of the same object represented by the same DN(Distinguished Name) in VMDir within the same replication cycle, can lead to replication issues ultimately resulting the VMDir in failure/unrecoverable state

Resolution

1. Set VMDir to Normal mode

/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL

2. Remove tombstone entries by following the steps in Troubleshooting and addressing accumulation of tombstones in a Platform Services Controller(embedded or external)

To manually remove tombstones in vCenter Server 6.0 Update 3 and later

  1. Connect to the Platform Services Controller using SSH and root credentials.
  2. Run this command to enable access the Bash shell:
shell.set --enabled true
  1. Type shell and press Enter.
  2. Run this command to create the all.txt file:
/opt/likewise/bin/ldapsearch -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' -b "cn=Deleted Objects,dc=vsphere,dc=local" -s sub "" -e 1.2.840.113556.1.4.417 dn | perl -p00e 's/?\n //g' | grep '^dn' > all.txt
  1. Run this command to remove all tombstone entries:
RULER="========"; while read line;do echo "$RULER$line$RULER"; time -p echo -e "$line\nchangetype: delete\n" | /opt/likewise/bin/ldapmodify -c -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' > /dev/null 2>/dev/null; done < all.txt | tee result.txt

Note:
  • If the default SSO domain name has been changed, then the dc value will need to be changed to match your environment.
  • If all nodes in the environment are Windows, a PSC appliance will need to be deployed to complete these steps. The appliance does not have to be connected to the current SSO domain to run the command above against a remote node.
  1. To verify the tombstone entries have been removed, run this command:
/opt/likewise/bin/ldapsearch -H ldap://PSC_FQDN -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'password' -b "cn=Deleted Objects,dc=vsphere,dc=local" -s sub "" -e 1.2.840.113556.1.4.417 dn| perl -p00e 's/?\n //g' | grep '^dn' | wc –l

3. Check the replication status using  Determining replication agreements and status with the Platform Services Controller (PSC)

Display the current replication status of the PSC and its partner nodes.

./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w Administrator_Password

4. Refrain from simultaneous operations, on the same DN ldap object. Keep the change(modification/addition/deletion) on one vCenter and wait for it to replicate on other VCs in the ELM.

Powered by Wolken Software