vCenter Server upgrade fails with VMDir replication is not working correctly during bundle precheck

Products

• VMware vCenter Server
• VCF Operations

Issue/Introduction

• The replication agreements were not removed completely, and the stale entries remained in the vCenter.
• Run the command to identify on each vCenter: /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u Administrator

root@VC1 [ ~ ]# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u Administrator
password:
Partner: VC2.domain.com
Host available:   Yes
Status available: Yes
My last change number:             1556148
Partner has seen my change number: 1479071
Partner is 369127 changes behind.

Partner: VC3.domain.com
Host available:   Yes
Status available: Yes
My last change number:             1556148
Partner has seen my change number: 1479071
Partner is 369127 changes behind.

Partner: VC4.domain.com
Host available:   No

• vCenter Server Preupgrade check result error:
• 

 

• Following /var/log/vmware/vmdird/vmdird-syslog.log there may be similar outputs

Environment

• vCenter 8.x
• VCF 5.x

Cause

If we can still see the replication agreement on the vCenter, it mean that the vCenter has not been properly decommissioned

Resolution

Unregister the Stale vCenter registry first using the cmsso-utils

1. Take powered-off Snapshots of all the vCenters either in the ELM or standalone

2. Run the below command to unregister stale vCenter.

cmsso-util unregister --node-pnid vCenterServer_System_Name --username administrator@your_domain_name --passwd 'vCenter_Single_Sign_On_password'

Where vCenter_Server_Appliance_System_Name is the FQDN or IP address of the vCenter Server Appliance that you want to decommission. Please enter the vCenter_Single_Sign_On_password in quotes.

For more information: https://knowledge.broadcom.com/external/article/316470

 

If the cmsso-utils is not removing completely, follow the Workaround below to manually delete the Entry:

•   Take powered-off Snapshots of all the vCenters either in the ELM or standalone, remove the replication agreements using the below command, and Reboot the vcenter.

* Determining replication agreements and status with the Platform Services Controller (PSC)
https://knowledge.broadcom.com/external/article/316504/determining-replication-agreements-and-s.html

./vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN -H PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password

 

If the removeagreement is not removing completely, follow the workaround below to use Jxplorer to manually delete:

LDAP

1. Download and install JXplorer from https://jxplorer.org/downloads/.
   Note: Java must be installed for JXplorer to function properly.
   
   
2. Open the JXplorer utility.
3. Click File > Connect and enter the following information:
   
   Host: vcenter.example.com
   Protocol: LDAPv3
   Port: 389
   Base DN: dc=vsphere,dc=local
   Level: User + Password
   User DN: cn=Administrator,cn=Users,dc=vsphere,dc=local
   
   Note: vsphere.local is the default naming convention for the SSO domain. If the environment has an SSO domain different to vsphere.local, replace as appropriate.

LDAPS

1. Download and install JXplorer from https://jxplorer.org/downloads/.
   Note: Java must be installed for JXplorer to function properly.
   
   
2. Open the JXplorer utility.
3. Click File > Connect and enter the following information.
   
   Host: vcenter.example.com
   Protocol: LDAPv3
   Port: 636
   Base DN: dc=vsphere,dc=local
   Level: SSL + User + Password
   User DN: cn=Administrator,cn=Users,dc=vsphere,dc=local

Once connected go to > Configuration >Sites > Lookupservice> Servers > vCenter Name >Replication Agreements > Right click and remove the stale agreement

Restart vCenter services once removed:

service-control --stop --all && service-control --start --all

Retry vCenter upgrade once more.