VMdir enters failure state after upgrading vCenter Server to 8.0 U1.

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0
• VMware vCenter Server

Issue/Introduction

• The vCenter Server started at version 6.5 or below, and has now been upgraded to 8.0U1.
• Messages in vCenter - /var/log/vmware/vmdird/vmdird-syslog.log show vmdir changing to an unrecoverable state following a reboot or service restart.

[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008367298304: _VmDirConsumePartner: Did not succesfully perform any updates after full pull. Moving vmdir to an unrecoverable state

[YYYY-MM-DDTHH:MM:SS] info vmdird  t@140008367298304: VmDir State (5)

[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008367298304: vdirReplicationThrFun: Replication has failed with unrecoverable error.

[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008241473280: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.

[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008241473280: VmDirSendLdapResult: Request (Search), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (10.10.10.10)

• There are also messages that indicate a replication conflict for the LegacyAliasMappings cn.

[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140008367298304: InternalDeleteEntry: VdirExecutePostDeleteCommitPlugins - code(9117)

[YYYY-MM-DDTHH:MM:SS] warning vmdird  t@140008367298304: ReplDeleteEntry/VmDirInternalDeleteEntry: 66 (Operation not allowed on non-leaf). DN: cn=LegacyAliasMappings,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,DC=vsphere,DC=local, first attribute: cn, it's meta data: '659195:2:abdefg-3891-435f-7afc-6b9636240bb3:20230429035650.714:426961'. NOT resolving this possible replication CONFLICT. For this object, system may not converge. Partner USN 0

 

Note: There is a small chance that the same replication conflict may occur for entries that are not LegacyAliasMapping. This will cause vmdir to go into the same failure mode. The action plan will be the same in these cases. 

• The domain functional level (DFL) of the vCenter is not "4". We will the observe the same entry in the VDT tool report.

To retrieve the DFL of vCenter, use the following command.

# /usr/lib/vmware-vmafd/bin/dir-cli domain-functional-level get

 

Environment

VMware vCenter Server 8.0.1

VMware vCenter Server 7.x

Cause

This occurs when the domain functional level of the vCenter has an unexpected value other than 4.

vCenter that has been upgraded since version 6.5 will have a DFL of 1. 

vCenter servers of version 7.0+ should have a DFL value of 4.

Resolution

This issue is resolved in vCenter Server 8.0 Update 2. To download, go to Download Broadcom products and software.

Workaround

1. Set the DFL of the affected node to 4 with the following command.
   
   /usr/lib/vmware-vmafd/bin/dir-cli domain-functional-level set --level 4 --login [email protected] --domain-name vsphere.local

Note: Update vsphere.local to match current SSO domain name.     
                                                                                                                    

2. Restart the vmdir service on all linked vCenter nodes.
   
   service-control --restart vmdird
   
   Note: Restart vmdir on all nodes only after updating the DFL of all the nodes in the ELM topology. Otherwise, vmdir will fail to start on the nodes which have a higher DFL than their partners.

Additional Information

 

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

• Product: vCenter
• Log File: vmdird-syslog.log
• Log Expression Check "ReplDeleteEntry/VmDirInternalDeleteEntry:" AND "Operation not allowed on non-leaf"

The workaround is also effective in scenarios where VDT reports the following error:

 [FAIL]    VMdir DFL Check

                        VMDIR Domain Functional Level is incorrect!

Note:

While running step 1 of the Workaround you may receive an error:

dir-cli failed, error= Invalid Domain Functional Level
Verify that level is valid for domain. 9129

If you encounter the above error while following the workaround, contact Broadcom Support and note this Article ID (318221) in the problem description. For more information, see Creating and managing Broadcom support cases.