vCenter Server Preupgrade check result error: "VMDir Replication between partners is not working"
search cancel

vCenter Server Preupgrade check result error: "VMDir Replication between partners is not working"

book

Article ID: 312122

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

vCenter Server Preupgrade check result error:

VMDir Replication between partners is not working.
Make sure vmdir service is reachable and started in partner nodes and this node before continuing.


 

.

Environment

VMware vCenter Server 6.7, 7.0 and 8.0

Cause

The vCenter upgrade fails due to Read-Only VMDir Mode.

In some cases, restoring a vCenter Server from snapshots without restoring the rest of the SSO domain nodes in Enhanced Linked Mode (ELM) can introduce inconsistencies in the VMDIR database, VMDir will change the state to Read-only to prevent potential data corruption amongst ELM partners when such restore operation is detected.

Resolution

As highlighted previously, the read-only mode is implemented as a protective measure against data corruption incidents, hence a definitive resolution is not available. However, this guide presents a workaround for scenarios where one or more ELM partners are detected in read-only mode.


Workaround

The steps to be followed are:

Note: Please be advised that we strongly recommend creating offline snapshots of the all the vCenter Servers in ELM setup prior to implementing the workaround. This measure will ensure that comprehensive rollback options are available if necessary.

Preparation:

  • Identify a healthy vCenter Server (either external or embedded) that you will use as a reference to retrieve VMDIR data. Remember the FQDN of this healthy vCenter Server and its root account password.                                                                                                                                                                    
    Note: If you're fixing an embedded node, choose another embedded node as the reference. If fixing an external node, choose an external node as your reference.

  • Keep a note of the administrator@your_domain password.
     

Steps to Repair:

  1. Snapshot Creation
    • Make offline snapshots of the entire ELM setup, including all VCs in the same vSphere Domain.
  2. Tool Download & Transfer
    • Download the `fix-psc-master.zip` tool attached to this KB.
    • Copy this tool to the problematic vCenter Server Node and then extract it
  3. Preliminary Check
    • Execute the pre-check command:
      ./fixpsc data --mode pre-check --healthy-psc-fqdn <healthy-node-fqdn>
    • Successful results will show similar to below snippets:
      INFO - vSphere SSO Domain Name: vsphere.local
      INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
      INFO - vSphere SSO Admin DN: cn=administrator,cn=users,dc=vsphere,dc=local
      .
      .
      INFO - Successfully executed script /usr/lib/repoint/taggingservice_component_script.py
      INFO - Conflict data, if any, can be found under /storage/domain-data/Conflict*.json
      INFO - Successfully finished <cis.component_data.DcComponentsPreCheck object at 0x7f39f5474850>
      INFO - pre-check completed
  4. Export Local Data
    • Use the export command:
      ./fixpsc data --mode export --healthy-psc-fqdn <healthy-node-fqdn>
    • Successful results will show similar to below snippets
      INFO - vSphere SSO Domain Name: vsphere.local
      INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
      INFO - vSphere SSO Admin DN: cn=administrator,cn=users,dc=vsphere,dc=local
      .
      .
      INFO - Successfully executed script /usr/lib/repoint/taggingservice_component_script.py
      INFO - Successfully finished <cis.component_data.DcExportCommand object at 0x7f24978e3b80>
      INFO - export completed
  5. VMDIR Rebuild
    • Execute the rebuild command:
      ./fixpsc rebuild --healthy-psc-fqdn <healthy-node-fqdn>
    • For reference, see the sample output below"
      INFO - Proceeding with fix-psc rebuild
      INFO - vSphere SSO Domain Name: vsphere.local
      INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
      .
      INFO - All End points un-registered successfully
      .
      INFO - Successfully created user machine in SSO
      .
      INFO - Successfully created user vsphere-webclient in SSO
      .
      INFO - Successfully created user vpxd in SSO
      .
      INFO - Successfully created user vpxd-extension in SSO
      .
      INFO - Creating service account...
      .
      INFO - Initializing ServiceAccount session...
             Done running command
  6. Import Data
    • Use the import command:
      ./fixpsc data --mode import --healthy-psc-fqdn <healthy-node-fqdn>
    • A successful operation can be checked against the sample snippets pasted below.
      INFO - vSphere SSO Domain Name: vsphere.local
      INFO - vSphere SSO Domain DN: dc=vsphere,dc=local
      INFO - vSphere SSO Admin DN: cn=administrator,cn=users,dc=vsphere,dc=local
      .
      .
      INFO - Starting all services                                                           ...
      INFO - All services Started.
      INFO - Successfully finished <cis.component_data.DcImportCommand object at 0x7f29884c9900>
      INFO - Successfully cleaned the storage directory
      INFO - import completed
  7. Service Registration Rectification (If required)
    • After running fixpsc, you can verify service registrations with the lsdoctor tool. Find detailed information at Using the 'lsdoctor' Tool.

Additional Information

The same error message during the upgrade of the vCenter will return when an old PSC was decommissioned but never got removed from the partners list.

Make sure no stale entries of old PSCs/vCenters are present in the directory, use vdcrepadmin for that: https://knowledge.broadcom.com/external/article/316504/ then if necessary use the cmsso-util to unregister it as per KB https://knowledge.broadcom.com/external/article?legacyId=2106736 

 

Attachments

fix-psc-master.zip get_app
Powered by Wolken Software