Enhanced Linked Mode (ELM) remains broken between the VCSA nodes

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

ELM may become "broken" due to many reasons, see some of these below -

• No offline snapshot was completed, and on restore, ELM is not in sync.
• Only 1 vCenter is restored to Snapshot/backup 
• Changes made to the networking of vCenter/s may disrupt replication

 

• In vCenter logs may see logs similar to the below: 

/var/log/vmware/vmdird/vmdird-syslog.log 

yyyy-mm-ddThh:mm:ss.Z.err vmdird  t@140008241473280: VmDirSendLdapResult: Request (Search), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in not in normal mode, not allowing outward replication.), (0) socket (10.10.10.10)

• After setting the VMDIR state back to normal, may experience issues with machine/computer account authentication errors; may see similar errors to those below -
• /var/log/vmware/vmdird/vmdird-syslog.log 

yyyy-mm-ddThh:mm:ss.Z err vmdird t@123456789123: Bind Request Failed (x.x.x.x) error 49: Protocol version: 3, Bind DN: ..., Method: SASL 12024-08-02T13:52:12.965554-05:00 err vmdird t@140245530842880: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

Environment

• VMware vCenter Server

 

Cause

• Due to the VMDIR state being in Read-only, replication is not functioning in the environment 
• This may also be due to changes made in the environment which caused mismatches in machine/computer account passwords between the vCenter server nodes in the SSO domain 

Resolution

To resolve the issue, please apply below steps: 

Note - Ensure to have valid offline snapshots and/or file-based backups completed of all nodes in the SSO domain before making any changes

Confirm the partner status of the vCenters in linked mode with the following command: 
 /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator 

Resolution Steps: 

1. Set  VMDIR state back to normal with the below steps:

   • SSH into the vCenter with root credentials and run the following commands: 

              /usr/lib/vmware-vmdir/bin/vdcadmintool  

              Then use option 5 to set the vmdir state to NORMAL

   • If the above fails proceed to step 2. 
2. Run the FixPSC script in KB: Fix PSC/vmdir inconsistencies using fixpsc python script
3. If there are still VMDIR inconsistencies please contact Broadcom Support for further assistance. 

         

Additional Information

Commands to run to check the replication status for all the VC in the ELM mode

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers -h localhost -u administrator

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

/usr/lib/vmware-vmafd/bin/dir-cli state get

hostname

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

service-control --status --all