Enhanced Linked Mode (ELM) Replication Broken between vCenter Server Appliance (VCSA) Nodes

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

An Enhanced Linked Mode (ELM) configuration in vCenter Server may become inconsistent or broken due to various reasons, This article outlines some of the common causes and considerations related to ELM failures:

• No offline snapshot was taken, and on restore, ELM is not in sync.
• Only 1 vCenter is restored to Snapshot/backup.
• Changes made to the networking of vCenter/s may disrupt replication.
• In vCenter server /var/log/vmware/vmdird/vmdird-syslog.log file the following entries will be seen: 

yyyy-mm-ddThh:mm:ss.Z.err vmdird  t@######241473280: VmDirSendLdapResult: Request (Search), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (##.##.##.##)

• After setting the VMDIR state back to normal, may experience issues with machine/computer account authentication errors; may see similar errors to those below in the /var/log/vmware/vmdird/vmdird-syslog.log file.

yyyy-mm-ddThh:mm:ss.Z err vmdird t@######789123: Bind Request Failed (##.##.##.##) error 49: Protocol version: 3, Bind DN: ..., Method: SASL 1YYYY-MM-DDTHH:MM:SS.965554-05:00 err vmdird t@#######30842880: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

• Due to the VMDIR state being in READ-ONLY or FAILURE, replication will not function in the environment.
• "Update Sequence Number (USN) mismatches" caused by restoring a single node from a snapshot without restoring the entire SSO domain.

Resolution

Prerequisites:

1. Ensure to have valid offline snapshots and/or file-based backups completed of all nodes in the SSO domain before making any changes.

2. Confirm the partner status of the vCenter servers in linked mode with the following command:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

Resolution Steps:

1. Set  VMDIR state back to normal with the below steps:

   • SSH into the vCenter with root credentials and run the following command:

/usr/lib/vmware-vmdir/bin/vdcadmintool

Use option 5 to set the vmdir state to NORMAL

1. • If the above vmdir state change encounter's a failure, proceed to step 2.

2. Run the FixPSC script in KB: Fix PSC/vmdir inconsistencies using fixpsc python script.

3. If there are still VMDIR inconsistencies contact Broadcom Support for further assistance.

Additional Information

Commands to run on vCenter Servers to check the replication status for all the vCenter Servers in the ELM mode:

• Check the list of servers: /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers -h localhost -u administrator
• Check the list of partners: /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator
• Check the partner status of vCenter Servers in ELM: /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator
• Check the state of the vmdir service: /usr/lib/vmware-vmafd/bin/dir-cli state get
• Check the hostname of the vCenter Server: hostname
• Check the PNID: /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
• Check the overall service status: service-control --status --all