Error: VMDir replication is not working correctly during SDDC Manager initiated vCenter Server upgrade
search cancel

Error: VMDir replication is not working correctly during SDDC Manager initiated vCenter Server upgrade

book

Article ID: 427637

calendar_today

Updated On:

Products

VMware vCenter Server VCF - MCOE - SDDC VMware vSphere ESXi

Issue/Introduction

The vCenter Server upgrade from SDDC Manager or vCenter Server update via the VAMI interface fails during the VCENTER_UPGRADE_PRECHECK stage with an error indicating a VMDir replication issue. 

Symptoms:

  • An SDDC Manager initiated vCenter Server upgrade fails during the VCENTER_UPGRADE_PRECHECK stage with the error: VMDir replication is not working correctly

  • Patching vCenter Server directly via the VAMI fails with the identical error: VMDir replication is not working correctly

  • The VMDir state returns a healthy status on all vCenter Server nodes when validated with the command: /usr/lib/vmware-vmafd/bin/dir-cli state get

  • Partner replication status appears healthy across all replicating vCenter Servers when verified using command: /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

  • Verification using the following command shows no stale or orphaned vCenter Server node: /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers -h localhost -u administrator 

  • The /var/log/vmware/applmgmt/patchRunner.log file records the following error:

"severity": "error",
"text": {
        "id": "vmdir.replicationerror.text",
        "localized": "VMDir replication is not working correctly",
        "translatable": "VMDir replication is not working correctly"
  • Log review of /var/log/vmware/vmdird/vmdird.log reveals mutual authentication failures:
    • vCenter Server where the upgrade/patch failed: 

YYYY:MM:DDTHH:MM:SS:t@####:ERROR: VmDirSafeLDAPBindEx to (ldap://Replicating_VC_FQDN:389) Failed. SRP(9234)

    • The replication partner vCenter Server referenced in the previous log entry:

YYYY:MM:DDTHH:MM:SS:t@####: ERROR: Bind Request Failed (####) error 49: Protocol version: 3, Bind DN: "cn=Replicating_VC_FQDN, ou=Domain Controllers, dc=vsphere, dc=local", Method: SASL

Environment

  • VMware Cloud Foundation 5.x

  • VMware vCenter Server 8.x

Cause

  • The machine account password used by the services to authenticate with the VMware Directory Service has become out of sync with the entry stored in the VMware Directory Service (VMDir) database, preventing successful authentication.

Resolution

Prerequsites: Before proceeding, take offline snapshot of all the vCenter Server in Enhanced Linked Mode. Refer: VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

Procedure:

  1. Reset the machine account password using the automated script as detailed in LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script 

  2. Re-initiate the vCenter Server upgrade/patch.