vCenter Server Preupgrade check result error: "VMDir Replication between partners is not working"

Products

VMware vCenter Server

Issue/Introduction

vCenter Server Preupgrade check result error:

VMDir Replication between partners is not working.
Make sure vmdir service is reachable and started in partner nodes and this node before continuing.

.

Environment

VMware vCenter Server 6.7, 7.0 and 8.0

Cause

The vCenter upgrade fails due to Read-Only VMDir Mode.

In some cases, restoring a vCenter Server from snapshots without restoring the rest of the SSO domain nodes in Enhanced Linked Mode (ELM) can introduce inconsistencies in the VMDIR database, VMDir will change the state to Read-only to prevent potential data corruption amongst ELM partners when such restore operation is detected.

Resolution

As highlighted previously, the read-only mode is implemented as a protective measure against data corruption incidents, hence a definitive resolution is not available. However, this guide presents a workaround for scenarios where one or more ELM partners are detected in read-only mode.

Workaround

The steps to be followed are:

Note: Please be advised that we strongly recommend creating offline snapshots of the all the vCenter Servers in ELM setup prior to implementing the workaround. This measure will ensure that comprehensive rollback options are available if necessary.

Preparation:

Identify a healthy vCenter Server (either external or embedded) that you will use as a reference to retrieve VMDIR data. Remember the FQDN of this healthy vCenter Server and its root account password.
Note: If you're fixing an embedded node, choose another embedded node as the reference. If fixing an external node, decide an external node as your reference.
Keep a note of the administrator@your_domain password.
Take note of the replication topology of the issue vCenter - this can be done with below command:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

Steps to Repair:

Snapshot Creation
- Make offline snapshots of the entire ELM setup, including all VCs in the same vSphere Domain.
Tool Download & Transfer
- Download the `fix-psc-master.zip` tool attached to this KB.
- Copy this tool to the problematic vCenter Server Node and then extract it
Preliminary Check
- Execute the pre-check command:
  ./fixpsc data --mode pre-check --healthy-psc-fqdn <healthy-node-fqdn>
- Successful results will show similar to below snippets:
  INFO - vSphere SSO Domain Name: vsphere.local
  INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
  INFO - vSphere SSO Admin DN: cn=administrator,cn=users,dc=vsphere,dc=local
  .
  .
  INFO - Successfully executed script /usr/lib/repoint/taggingservice_component_script.py
  INFO - Conflict data, if any, can be found under /storage/domain-data/Conflict*.json
  INFO - Successfully finished <cis.component_data.DcComponentsPreCheck object at 0x7f39f5474850>
  INFO - pre-check completed
Export Local Data
- Use the export command:
  ./fixpsc data --mode export --healthy-psc-fqdn <healthy-node-fqdn>
- Successful results will show similar to below snippets
  INFO - vSphere SSO Domain Name: vsphere.local
  INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
  INFO - vSphere SSO Admin DN: cn=administrator,cn=users,dc=vsphere,dc=local
  .
  .
  INFO - Successfully executed script /usr/lib/repoint/taggingservice_component_script.py
  INFO - Successfully finished <cis.component_data.DcExportCommand object at 0x7f24978e3b80>
  INFO - export completed
VMDIR Rebuild
- Execute the rebuild command:
  ./fixpsc rebuild --healthy-psc-fqdn <healthy-node-fqdn>
- For reference, see the sample output below"
  INFO - Proceeding with fix-psc rebuild
  INFO - vSphere SSO Domain Name: vsphere.local
  INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
  .
  INFO - All End points un-registered successfully
  .
  INFO - Successfully created user machine in SSO
  .
  INFO - Successfully created user vsphere-webclient in SSO
  .
  INFO - Successfully created user vpxd in SSO
  .
  INFO - Successfully created user vpxd-extension in SSO
  .
  INFO - Creating service account...
  .
  INFO - Initializing ServiceAccount session...
  Done running command
Import Data
- Use the import command:
  ./fixpsc data --mode import --healthy-psc-fqdn <healthy-node-fqdn>
- A successful operation can be checked against the sample snippets pasted below.
  INFO - vSphere SSO Domain Name: vsphere.local
  INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
  INFO - vSphere SSO Admin DN: cn=administrator,cn=users,dc=vsphere,dc=local
  .
  .
  INFO - Starting all services ...
  INFO - All services Started.
  INFO - Successfully finished <cis.component_data.DcImportCommand object at 0x7f29884c9900>
  INFO - Successfully cleaned the storage directory
  INFO - import completed
Service Registration Rectification (If required)
- After running fixpsc, you can verify service registrations with the lsdoctor tool. Find detailed information at Using the 'lsdoctor' Tool.
Recreate necessary replication agreements to restore ring topology
- A replication agreement will be automatically created between the vCenter the fixpsc script has been run on and the targeted healthy vCenter
- To restore replication agreements to any other vCenters that the issue vCenter previously had - use below command:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f createagreement -2 -h localhost -H <TARGET_VC> -u administrator

Additional Information

The same error message during the upgrade of the vCenter will return when an old PSC was decommissioned but never got removed from the partners list.

Make sure no stale entries of old PSCs/vCenters are present in the directory, use vdcrepadmin for that: Determining replication agreements and status with the Platform Services Controller (PSC) then if necessary use the cmsso-util to unregister it as per KB Using the cmsso command to unregister vCenter with External PSC or vCenter with Embedded PSC from Single Sign-On

Determining replication agreements and status with the Platform Services Controller (PSC)

Attachments

fix-psc-master.zip get_app