VMdird service fails to start with error : "An error occurred while starting service 'vmdird'" | "Server is at a higher functional level (4) than partner" "(0) and cannot perform at a lower level"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms: VMDIRD service fails to start on ELM setup

Service Start fails with error : 
 "An error occurred while starting service 'vmdird'"

vmdird-syslog.log : 

YYYY-MM-DD HH-MM-SS err vmdird  t@139872566388544: Server is at a higher functional level (4) than partner (vcenterdr.example.com)(0) and cannot perform at a lower level.

journalctl -xe : 

MONTH 20 18:35:54 vcenterprod.example.com vmdird[34790]: t@140052223379264: VmDirKrbInit, REALM (VSPHERE.LOCAL)
MONTH 20 18:35:54 vcenterprod.example.com vmdird[34790]: t@140052223379264: Server is at a higher functional level (4) than partner (vcenterdr.example.com)(0) and cannot perform at a lower level.
MONTH 20 18:35:54 vcenterprod.example.com lwsmd[1990]: Service died: vmdir (restarted 2 times in 0 seconds)

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

During the startup of the VMdir, it tries to contact the partner nodes for DFL (domain functional level) by reading the "dc=vsphere,dc=local" entry using the Machine account and the machine account will be missing from DCAdmins. 
Since the Machine account doesn't have the required privileges/missing, ldapsearch will simply return zero entries and DFL value become 0 and thus service start fails. 
 

Resolution

In this example: We have vcenterprod.example.com & vcenterdr.example.com as vCenter nodes in ELM where vcenterprod.example.com is down. 

To resolve this issue, find the partner from logs (vmdird or journalctl -xe) 
If the customer environment allows to use jXplorer.
Connect to the partner node. In this example - its vcenterdr.example.com :  YYYY-MM-DDT18:35:54.810901+00:00 err vmdird  t@140052223379264: Server is at a higher functional level (4) than partner (vcenterdr.example.com)(0) and cannot perform at a lower level.


Builtin - DCAdmins

 

If we cannot use jXplorer, from partner node ssh run the below command to get the details. 

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' member

As we can see the node (vcenterprod.example.com) is missing here.
We can add the missing node as a member and Submit the changes and try starting vmdird.

If the environment has restrictions on using Jxplorer then follow the below steps :

SSH to partner node ( from logs)

cd /var/tmp

create a file with below contents. (Assuming SSO domain as default vsphere.local)

vi node.ldif

dn: cn=DCAdmins,cn=Builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: cn=vcenterprod.example.com,ou=Domain Controllers,dc=vsphere,dc=local

Save the file.

Run :

/opt/likewise/bin/ldapadd -h localhost -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -w 'SSOPWD' -f node.ldif

Start vmdird service.