VMdir enters failure state after upgrading vCenter Server to 8.0 U1.
search cancel

VMdir enters failure state after upgrading vCenter Server to 8.0 U1.

book

Article ID: 318221

calendar_today

Updated On:

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0 VMware vCenter Server

Issue/Introduction

  • The vCenter Server started at version 6.5 or below, and has now been upgraded to 8.0U1.
  • Messages in vCenter - /var/log/vmware/vmdird/vmdird-syslog.log show vmdir changing to an unrecoverable state following a reboot or service restart.
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008367298304: _VmDirConsumePartner: Did not succesfully perform any updates after full pull. Moving vmdir to an unrecoverable state
[YYYY-MM-DDTHH:MM:SS] info vmdird  t@140008367298304: VmDir State (5)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008367298304: vdirReplicationThrFun: Replication has failed with unrecoverable error.
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008241473280: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008241473280: VmDirSendLdapResult: Request (Search), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (10.10.10.10)
  • There are also messages that indicate a replication conflict for the LegacyAliasMappings cn.
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008367298304: InternalDeleteEntry: VdirExecutePostDeleteCommitPlugins - code(9117)
[YYYY-MM-DDTHH:MM:SS] warning vmdird  t@140008367298304: ReplDeleteEntry/VmDirInternalDeleteEntry: 66 (Operation not allowed on non-leaf). DN: cn=LegacyAliasMappings,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,DC=vsphere,DC=local, first attribute: cn, it's meta data: '659195:2:abdefg-3891-435f-7afc-6b9636240bb3:20230429035650.714:426961'. NOT resolving this possible replication CONFLICT. For this object, system may not converge. Partner USN 0
 
Note: There is a small chance that the same replication conflict may occur for entries that are not LegacyAliasMapping. This will cause vmdir to go into the same failure mode. The action plan will be the same in these cases. 
  • The domain functional level (DFL) of the vCenter is not "4".
To retrieve the DFL of vCenter, use the following command.

# /usr/lib/vmware-vmafd/bin/dir-cli domain-functional-level get

 

Environment

VMware vCenter Server 8.0.1
VMware vCenter Server 7.x

Cause

This occurs when the domain functional level of the vCenter has an unexpected value other than 4.
vCenter that has been upgraded since version 6.5 will have a DFL of 1. 
vCenter servers of version 7.0+ should have a DFL value of 4.

Resolution

This issue is resolved in vCenter Server 8.0 Update 2. To download, go to Download Broadcom products and software.

Workaround

  1. Set the DFL of the affected node to 4 with the following command.

    /usr/lib/vmware-vmafd/bin/dir-cli domain-functional-level set --level 4 --login [email protected] --domain-name vsphere.local

    Note: Update vsphere.local to match current SSO domain name.     
                                                                                                                     
  1. Restart the vmdir service on all linked vCenter nodes.

    service-control --restart vmdird

    Note: Restart vmdir on all nodes only after updating the DFL of all the nodes in the ELM topology. Otherwise, vmdir will fail to start on the nodes which have a higher DFL than their partners.

Additional Information

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

  • Product: vCenter
  • Log File: vmdird-syslog.log
  • Log Expression Check "ReplDeleteEntry/VmDirInternalDeleteEntry:" AND "Operation not allowed on non-leaf"
Powered by Wolken Software