Determining replication agreements and status with the Platform Services Controller (PSC)
search cancel

Determining replication agreements and status with the Platform Services Controller (PSC)

book

Article ID: 316504

calendar_today

Updated On:

Products

VMware vCenter Server 6.0 VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

This article provides information on using the vdcrepadmin command-line interface (CLI) for reviewing the vSphere domain, the Platform Services Controllers (PSC), and checking the replication agreements configured and replication status within an environment.
 
Although the utility can be used for other operations, at this time, only what is documented must be executed by technical support staff and end users.
 
The vdcrepadmin CLI is located in these directories depending on the PSC distribution used:
  • Appliance: /usr/lib/vmware-vmdir/bin

    Note: This requires BASH Shell access to the appliance. For steps on switching from the appliance shell to the BASH shell, see Toggling the vCenter Server Appliance 6.x default shell.
     
  • Windows: "%VMWARE_CIS_HOME%"\vmdird\

    By default, the environment variable will be: C:\Program Files\VMware\vCenter Server\

Resolution

Process to determine replication agreements and status with the Platform Services:

Use the below parameters using the vdcrepadmin command-line interface (CLI) for reviewing the vSphere domain to check replication agreements configured and replication status within the environment:

  • showservers - Displays all of the PSCs within a vSphere domain.
  • showpartners - Displays the current partnerships from a single PSC within a vSphere domain.
  • showpartnerstatus - Displays the current replication status of a PSC and any of the replication partners of the PSC.
  • createagreement - Allows for creation of additional replication agreements between PSCs within a vSphere domain.
  • removeagreement - Allows for removal of additional replication agreements between PSCs within a vSphere domain.

Note: For vCenter 6.7 the showfederationstatus command was added to support additional debugging function.
 

Using the showservers parameter

  1. Remote access into the windows Platform Services Controller or SSH into the Platform Services Controller appliance.
  2. Change directories to the vmdird folder.

    Appliance: cd /usr/lib/vmware-vmdir/bin
    Windows: cd "%VMWARE_CIS_HOME%"\vmdird\
     
  3. Show all PSCs in the vSphere domain.

    ./vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w Administrator_Password
Note: Enter the administrator_password in between quotes. 

./vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w 'Administrator_Password'

If there is an error message such as -bash: ##### event not found, ensure that the use of the ' character not " is used to encapsulate the password.
 
Example
./vdcrepadmin -f showservers -h psc1.example.local -u administrator -w password

Example Output
cn=psc1.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc2.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc4.example.local,cn=Servers,cn=East,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc3.example.local,cn=Servers,cn=East,cn=Sites,cn=Configuration,dc=vsphere,dc=local

From the output, determine the required information.

Example
cn=psc1.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local
    • The Platform Services Controller is named psc1.example.local.
    • The Platform Services Controller is located in the home site within the vSphere domain.
    • To determine this information, review the full output:
      • The Platform Services Controllers psc1.* and psc2.* are located within the site Home.
      • The Platform Services Controllers psc3.* and psc4.* are located within the site East.

Using the showpartners parameter

  1. Remote access into the windows Platform Services Controller or SSH into the Platform Services Controller appliance.
  2. Change directories to the vmdird folder.

    Appliance: cd /usr/lib/vmware-vmdir/bin

    Windows: cd "%VMWARE_CIS_HOME%"\vmdird\
     
  3. Display the current partnership from the specified PSC:

Note: For 6.7, use: vdcrepadmin -f showfederationstatus -h localhost -u administrator -w Administrator_Password

./vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password

Example

./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password

This command provides PSC specified by -h parameter, partnership.

Example
./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
ldap://psc2.example.local

    4. Use the PSC from Step 3 to map out the topology of the current vSphere domain by re-running this command against each of the PSCs in order to determine all of the partnerships.

        Example
        Note: Use the showservers parameter to get a list of all of the PSCs in the domain.

./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
ldap://psc2.example.local

./vdcrepadmin -f showpartners -h psc2.example.local -u administrator -w password
ldap://psc1.example.local
ldaps://psc3.example.local

./vdcrepadmin -f showpartners -h psc3.example.local -u administrator -w password
ldap://psc4.example.local
ldaps://psc2.example.local

./vdcrepadmin -f showpartners -h psc4.example.local -u administrator -w password
ldap://psc3.example.local
 
From this series of output, it can be determined that:
    • PSC1.* has a replication partnership with PSC2*
    • PSC2.* has a replication partnership with both PSC1.* and PSC3.*
    • PSC3.* has a replication partnership with both PSC2.* and PSC4.*
    • PSC4.* has a replication partnership with PSC3.*
This illustrates that this environment was installed in an in-line fashion, with each PSC installed against the previous PSC, rather than a hub-and-spoke fashion where all of the PSCs would terminate to a central PSC.
 

Using the showpartnerstatus parameter:

Note: This CLI is limited to execution only against the local PSC. Using the command to query the replication status from one PSC to a different PSC is not yet supported.
  1. Remote access into the windows Platform Services Controller or SSH into the Platform Services Controller appliance.
  2. Change directories to the vmdird folder.
    Appliance: cd /usr/lib/vmware-vmdir/bin
    Windows: cd "%VMWARE_CIS_HOME%"\vmdird\
     
  3. Display the current replication status of the PSC and its partner nodes.

    ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password

    This command displays the current replication partner of the PSC and also the current replication status between the two nodes.

    Example
    psc3:~ # /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password
    Partner: psc4.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 9502
    Partner has seen my change number: 9502
    Partner is 0 changes behind.

    Partner: psc2.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 9502
    Partner has seen my change number: 9502
    Partner is 0 changes behind.
From this output, the following information can be determined for the current node (psc3.example.local):
    • There are two replication agreements, one with psc2.* and another with psc4.*C
    • Connection to the two partner PSCs is successful as is the ability to query their status
    • Sync is current with all replication partners within the environment
    • The current Update Sequence Number (USN) value for this PSC is 9502.
Repeat this operation on the other PSCs in the vSphere domain to check all replication status.
 
If the replication continues to fail on the customers environment, review the /var/log/vmware/vmdird/vmdird-syslog.log or "%VMWARE_LOG_DIR%"\vmdird\vmdird-syslog.log file for details. This provides all information related to replication status and the objects that are replicated.
 

Using the createagreement parameter

WARNING

Before making linked mode changes ensure that proper snapshots of the SSO domain or file-based backups of each vCenter via the VAMI have been taken. Refer to KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice for further instructions. Failure to perform these steps can lead to replication problems across the VC databases. 

Note: This cannot be used to create replication agreements between disparate (separate) vSphere domains.
  1. Remote access into the windows Platform Services Controller or SSH into the Platform Services Controller appliance.
  2. Change directories to the vmdird folder.
    Appliance: cd /usr/lib/vmware-vmdir/bin
    Windows: cd "%VMWARE_CIS_HOME%"\vmdird\
     
  3. Display the current partnership from the specified PSC.

    ./vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password

    This provides PSC specified by -h parameter, partnership.

    Example
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
    ldap://psc2.example.local
     
  4. Map out the topology of the current vSphere domain by re-running this command against each of the PSCs in order to determine all of the partnerships.

    Example
    Note: Use the showservers parameter to get a list of all of the PSCs in the domain.

    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
    ldap://psc2.example.local

    ./vdcrepadmin -f showpartners -h psc2.example.local -u administrator -w password
    ldap://psc1.example.local
    ldaps://psc3.example.local

    ./vdcrepadmin -f showpartners -h psc3.example.local -u administrator -w password
    ldap://psc4.example.local
    ldaps://psc2.example.local

    ./vdcrepadmin -f showpartners -h psc4.example.local -u administrator -w password
    ldap://psc3.example.local
     
  5. With the topology defined, generate a new replication agreement to create a Ring Topology. Using the environment in this article as a model, generate additional replication agreements between psc1.* and psc4.*.

    However, in larger environments with more PSCs or environments that have grown, replication agreements may need to be removed or adjusted accordingly. For the latter use case, consult the Adding Platform Services Controllers to grow your vSphere Domain and updating replication agreements in the Related Information section of this article for more guidance.
     
  6. Create a new replication agreement between PSCs to generate a ring topology.

    ./vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f createagreement -2 -h psc1.example.local -H psc4.example.local -u Administrator -w password

    Note: PSC Names in the syntax is case sensitive. The vSphere Client will display the name in lower case. Be cautious of case sensitivity.

    Use the following as an example as a visual representation of the recommended outcome.

  7. After completion, repeat Step 4 to confirm that the environment is now in a ring topology.

    Note: Due to replication time, it may take a few seconds to minutes for a complete ring topology to be configured.

Using the removeagreement parameter

Note: This command cannot be utilized to completely remove all replication partnerships from a PSC. At a minimum, a PSC requires a single replication agreement depending on the topology.
  1. Remote access into the windows Platform Services Controller or SSH into the Platform Services Controller appliance.
  2. Change directories to the vmdird folder.
    Appliance: cd /usr/lib/vmware-vmdir/bin
    Windows: cd "%VMWARE_CIS_HOME%"\vmdird\
     
  3. Display the current partnership from the specified PSC.
    ./vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password

    This provides PSC partnership specified by the -h parameter.

    Example
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
    ldap://psc2.example.local
     
  4. Map out the topology of the current vSphere domain by re-running this command against each of the PSCs in order to determine all of the partnerships.

    Example

    Note: Use the showservers parameter to get a list of all of the PSCs in the domain. In the example below, we have a ring topology.
     
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
    ldap://psc2.example.local
    ldaps://psc4.example.local
     
    ./vdcrepadmin -f showpartners -h psc2.example.local -u administrator -w password
    ldap://psc1.example.local
    ldaps://psc3.example.local
     
    ./vdcrepadmin -f showpartners -h psc3.example.local -u administrator -w password
    ldap://psc4.example.local
    ldaps://psc2.example.local
     
    ./vdcrepadmin -f showpartners -h psc4.example.local -u administrator -w password
    ldap://psc3.example.local
    ldaps://psc1.example.local

  5. With the topology defined, evaluate the removal of replication agreement. Using the environment in this article as a model, remove the replication agreements between the vCenters in order to revert the four node topology into an inline layout from its previous ring layout: psc1.* and psc4.*
     
  6. Remove the existing replication agreement between PSCs.
    ./vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN -H PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f removeagreement -2 -h psc1.example.local -H psc4.example.local -u Administrator -w password

    This will revert the four node topology into an inline layout from its previous ring layout.
     
  7. Repeat this operation to remove any additional replication agreements between PSCs.

ELM.SH (An automated method for checking all the above commands in a single report.)

Running each vdcrepadmin command requires the user to re-enter the password. As opposed to entering in the password for each of these 6 commands, multiplied by the amount of vCenter's in a SSO domain. This script automates the checking of all replication commands as well as the VMDIRD state. This script makes no changes, but instead runs the above replication agreement commands to read the present status and health of the vCenter's in a given SSO domain.

Usage

  1. Create a file on the vCenter

    vi /tmp/elm.sh

  2. Copy and paste the below bash script contents to the file.

    #!/bin/bash
read -p "Enter [email protected] password: " -s pw
clear
echo ">>>>>>>>>>>>>>>>>>>>>>"
echo "PNID"
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
echo ">>>>>>>>>>>>>>>>>>>>>>"
echo "                      "
echo "########################################"
echo "ShowServers"
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showservers -h localhost -u administrator -w $pw
echo "                      "
echo "                      "
echo "########################################"
echo "ShowPartners"
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator -w $pw
echo "                      "
echo "                      "
echo "########################################"
echo "ShowPartnerStatus"
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w $pw
echo "                      "
echo "                      "
echo "########################################"
echo "ShowFederationStatus"
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showfederationstatus -h localhost -u administrator -w $pw
echo "                      "
echo "                      "
echo "########################################"
echo "VMDIR State"
/usr/lib/vmware-vmafd/bin/dir-cli state get --password $pw
  3. Save the file

    :wq!

  4. Add executable permissions on the elm.sh file

    chmod +x elm.sh

  5. Run the file to print the replication health report.

    ./elm.sh

    Note:     If there is a bad interpreter message as a result of copying between Windows and Linux, use the following command on the vCenter to correct the formatting to allow the script to run successfully:

    sed -i 's///' elm.sh

Additional Information

Verifying the replication after installing vCenter Server or creating a user/group

Using the showpartnerstatus CLI, monitor the current replication status across the environment. There are times in which replication may not be functioning properly and replication data from one PSC may not be reaching another PSC.

After modifying objects within the PSC, either via creation, deletion or actual modification, these changes are replicated to the PSCs after 30 seconds. When multiple PSCs are daisy chained together (in-line), this can result in a wave-replication in which PSCs in the chain will request an update, implement the update, and then its partner node will perform the same, eventually converge on the same replica data across the same vSphere domain.
 
After creating and deleting an object via the vSphere Web Client UI (user), observe these objects deletion permeate the domain:
  1. User creation and deletion on psc1.*
    psc1:~ # cd /usr/lib/vmware-vmdir/bin ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password
    Partner: psc2.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 12227
    Partner has seen my change number: 12222
    Partner is 5 changes behind.
     
  2. Repeating this command from the psc2.* simultaneously:

    Before modification from psc1.* is picked up
    psc2:~ # cd /usr/lib/vmware-vmdir/bin ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password
    Partner: psc1.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 14539
    Partner has seen my change number: 14539
    Partner is 0 changes behind.

    Partner: psc3.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 14534
    Partner has seen my change number: 14534
    Partner is 5 changes behind.

    After modification from psc1.* is picked up
    psc2:~ # cd /usr/lib/vmware-vmdir/bin ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password
    Partner: psc1.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 14539
    Partner has seen my change number: 14539
    Partner is 0 changes behind.

    Partner: psc3.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 14539
    Partner has seen my change number: 14539
    Partner is 0 changes behind.
     
  3. Repeating this command from the psc3.* simultaneously:

    Before modification from psc2.* is picked up
    psc3:~ # cd /usr/lib/vmware-vmdir/bin ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password
    Partner: psc4.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 9530
    Partner has seen my change number: 9523
    Partner is 7 changes behind.

    Partner: psc2.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 9530
    Partner has seen my change number: 9530
    Partner is 0 changes behind.

    After modification from psc2.* is picked up

    psc3:~ # cd /usr/lib/vmware-vmdir/bin ./vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password
    Partner: psc4.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 9530
    Partner has seen my change number: 9530
    Partner is 0 changes behind.

    Partner: psc2.example.local
    Host available: Yes
    Status available: Yes
    My last change number: 9530
    Partner has seen my change number: 9530
    Partner is 0 changes behind.

Adding Platform Services Controllers to grow the vSphere Domain and updating replication agreements

Using the showservers, createagreements, and removeagreements CLI, the ability to update the vSphere Domain's replication topology as additional Platform Services Controllers are added to the environment.
Using the model defined throughout this article using a four-node environment, add an additional pair of Platform Services Controllers in a new site, bringing the environment up to six total, and then update the replication topology.
  1. Remote access into the windows Platform Services Controller or SSH into the Platform Services Controller appliance.
  2. Change directories to the vmdird folder:
    Appliance: cd /usr/lib/vmware-vmdir/bin
    Windows: cd "%VMWARE_CIS_HOME%"\vmdird\
     
  3. Show all PSCs in the vSphere domain.
    ./vdcrepadmin -f showservers -h PSC_FQDN -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f showservers -h psc1.example.local -u administrator -w password

    Example Output
cn=psc1.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc2.example.local,cn=Servers,cn=home,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc4.example.local,cn=Servers,cn=East,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc3.example.local,cn=Servers,cn=East,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc5.example.local,cn=Servers,cn=West,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=psc6.example.local,cn=Servers,cn=West,cn=Sites,cn=Configuration,dc=vsphere,dc=local
From the output, determine the required information. In this example:
  • The Platform Services Controllers psc1.* and psc2.* are located within the site Home.
  • The Platform Services Controllers psc3.* and psc4.* are located within the site East.
  • The Platform Services Controllers psc5.* and psc6.* are located within the site West.
  1. Display the current partnership from the specified PSC.
    ./vdcrepadmin -f showpartners -h PSC_FQDN -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password

    This command provides PSC partnership specified by the -h parameter.

    Example
    ./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
    ldap://psc2.example.local
     
  2. Use the PSC from Step 4 to map out the topology of the current vSphere domain by re-running this command against each of the PSCs in order to determine all of the partnerships.

    Example
    Note: Use the showservers parameter to get a list of all of the PSCs in the domain.
./vdcrepadmin -f showpartners -h psc1.example.local -u administrator -w password
ldap://psc2.example.local
ldaps://psc4.example.local
 
./vdcrepadmin -f showpartners -h psc2.example.local -u administrator -w password
ldap://psc1.example.local
ldaps://psc3.example.local
 
./vdcrepadmin -f showpartners -h psc3.example.local -u administrator -w password
ldap://psc4.example.local
ldaps://psc2.example.local
 
./vdcrepadmin -f showpartners -h psc4.example.local -u administrator -w password
ldap://psc3.example.local
ldaps://psc1.example.local
ldaps://psc5.example.local
 
./vdcrepadmin -f showpartners -h psc5.example.local -u administrator -w password
ldap://psc4.example.local
ldaps://psc6.example.local
 
./vdcrepadmin -f showpartners -h psc6.example.local -u administrator -w password
ldap://psc5.example.local
From this series of output, the following information can be determined:
  • psc1.* has a replication partnership with both psc2* and psc4.*
  • psc2.* has a replication partnership with both psc1.* and psc3.*
  • psc3.* has a replication partnership with both psc2.* and psc4.*
  • psc4.* has a replication partnership with both psc3.* , psc1.* and psc5.*
  • psc5.* has a replication partnership with both psc4.* and psc6.*
  • psc6.* has a replication partnership with psc5.*
  1. With the update topology defined, accounting for the addition of two new PSCs, generate a new replication agreement to update the ring topology. Using the environment in this article as a model:

    Generate additional replication agreements between:
    psc1.* and psc6.*

    Remove the old replication agreements between:
    psc1.* and psc4.*
     
  2. Create a new replication agreement between PSCs to update the ring topology:
    ./vdcrepadmin -f createagreement -2 -h Source_PSC_FQDN -H New_PSC_FQDN_to_Replicate -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f createagreement -2 -h psc1.example.local -H psc6.example.local -u Administrator -w password
     
  3. Remove an existing replication agreement between PSCs:
    ./vdcrepadmin -f removeagreement -2 -h Source_PSC_FQDN -H PSC_FQDN_to_Remove_from_Replication -u administrator -w Administrator_Password

    Example
    ./vdcrepadmin -f removeagreement -2 -h psc1.example.local -H psc4.example.local -u Administrator -w password

    Use the following as a example as visual representation of the recommended outcome:

  1. After completion, repeat Step 4 to confirm that the environment is now a ring topology with all 6 nodes.

    Note: Due to replication time, it may take a few seconds to minutes for a complete ring topology to be configured.
Powered by Wolken Software