vCenter Server Replication Status Change Alarm in the vCenter Server UI

Products

• VMware vCenter Server

Issue/Introduction

vCenter servers in Enhanced Linked mode show the following error in the UI:

vCenter server replication status change alarm

Checking Replication status with the following command will show that the partner is available = no and will be multiple changes behind:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator

Example output:

Host available:   Yes
Status available: No

My last change number:             102676
Partner has seen my change number: 99887
Partner is 2789 changes behind.

Additionally, manually Changing the vmdir status from standalone to normal does not stick and reverts following a reboot/power cycle of the vCenter.

 

NOTE: In the event of an output that shows a consistently large Partner is #### changes behind number or Host available: No please see the FixPSC section under Resolution in this article.

Environment

VMware vCenter Server

Cause

Upon checking the vmdir status with the following command, you will find that the status is set to STANDALONE:

• You can check the VMDIR state with option 6 of following command:

  /usr/lib/vmware-vmdir/bin/vdcadmintool

If you change the VMDIR mode to NORMAL it will stay that way until a subsequent restart of the vmdir service or the VCSA is rebooted, at which point the VMDIR mode will be switch back into STANDALONE mode.

There can be multiple causes for this issue to occur including for example:

• Status of vmdir can be left in standalone mode following vCenter patch or upgrade.

Resolution

Take offline snapshots of all vCenters in ELM before making changes.

NOTE: If the VMDIR mode is set to Failure (5) try to restart all services of the affected vCenter Server. Once the services are restarted check the VMDIR mode again, if it doesn't change to NORMAL or still the showpartnerstatus shows partner status available as No, proceed with the following steps.

1. Run the following command on the affected vCenter:
   

   /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir\]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif"

   For 8.0 U3 and above versions:
   

   /opt/likewise/bin/lwregshell set_value '[HKEY_THIS_MACHINE\Services\vmdir]' "Arguments" "/usr/lib/vmware-vmdir/sbin/vmdird -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif -L /var/log/vmware/vmdird/vmdird.log"

2. Followed by the following two commands:
   

   /opt/likewise/bin/lwsm refresh
   service-control --stop vmdird && service-control --start vmdird

 

FixPSC:

• The FixPSC script can be used to manually re-sync a vCenter's VMDIR store with another in which it is supposed to be linked with.
  • Access the script and supporting documentation here: Fix PSC/vmdir inconsistencies using fixpsc python script

• To use the script to re-sync vCenters: 
  1. Take Powered off snapshots of the vCenter Server(s) which you are running the commands below on.
  2. Upload the FixPSC script to a directory (/tmp is preferred) using the SCP/SFTP tool of your preference.
  3. Run the command below to resync your vCenter with its partner: 
     • ./fixpsc rebuild --healthy-psc-fqdn <Other_vCenter_FQDN>
  4. When prompted, enter your SSO admin password. 
  5. The process will take several minutes to complete. During this time, the vCenter will copy files from the partner vCenter and restart the VMDIR services. 
  6. Re-check the Partner Status using the command below on both vCenters:
     • /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator