Using vmdir_tool.py to identify vmdir/ELM replication inconsistencies

Products

VMware vCenter Server

Issue/Introduction

VMDIR Tool is a tool used to address issues with data stored in the ELM database. The tool can be used to detect and correct problems that could cause failures in topology changes (converge, repoint, etc.), upgrades, or failures incurred as a result of maintenance (e.g. incorrectly applying new SSL certificates). This article will outline its functions and use.

VMdird replication issues.
When querying the vmdir state using vdcadmintool, you get a 'Null' or 'Read Only' state
Querying replication status shows 'Status available: No"
Other symptoms, such as:
- User/Group changes are not propagating to other vCenters in the SSO domain
- Cannot repoint, upgrade, converge
- Cannot register other vSphere products

Error messages in the vmdird-syslog.log (or vmdird.log) file located in /var/log/vmware/vmdird/

err vmdird t@139841484338944: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
err vmdird t@140609914369792: UpdateServerObject: InternalModifyEntry failed. Error code: 53, Error string: Server in read-only mode

Environment

vCenter server 7.x
vCenter server 8.x

Cause

When vCenter servers in ELM are restored from snapshots improperly, inconsistencies in the VMDIR database are introduced. The database will attempt to repair data gaps, but this may not always be successful.
This tool can help identify and troubleshoot such scenarios.

Resolution

Tool Use: vmdir-tool
This tool should primarily be used to identify ELM database inconsistencies.
Note: Only the read functions integrityCheck, replicationValidation, and getTopology options should be used. Other options may be destructive.
Tool directory: cd /usr/lib/vmware-vmdir/vmdir-tool/
1. View the help menu: python /usr/lib/vmware-vmdir/vmdir-tool/vmdir_tool.py --help
  usage: vmdir_tool.py [-h] [--hostname HOSTNAME] [--getTopology] [--integrityCheck] [--hostOnly] [--replicationValidation]
  [--uptodateVectorCheck] [--restoreVmdir] [--machineAccount]
  
  VMDir Debug Tool
  
  options:
  -h, --help show this help message and exit
  --hostname HOSTNAME Hostname or FQDN of the node
  --getTopology Store Topology information of the nodes in /var/log/vmware/vmdir/topology_info.json
  --integrityCheck Check data inconsistency between all nodes. Generated logs can be found in /var/log/vmware/vmdir. NOTE:The output
  of this check is not guranteed to be complete always in a log support bundle. In such cases it must be run
  separately to collect the complete output.
  --hostOnly Used with integrityCheck flag to perform checks only from the specified node for hostname
  --replicationValidation
  Validate replication between all the nodes
  --uptodateVectorCheck
  Validate consistency between vmdir data & up-to-date vector
  --restoreVmdir Restore Vmdir
  --machineAccount
Initiate an integrity check to compare database entries between nodes:

python /usr/lib/vmware-vmdir/vmdir-tool/vmdir_tool.py --hostname $HOSTNAME --integrityCheck --machineAccount

Note: The '$HOSTNAME' value will be automatically expanded to the current vCenter hostname. The value does not need changing.
Additionally, the --machineAccount parameter will run the tool without having to provide a username and password.
Tool results:

All tool results and logs are in the directory: /var/log/vmware/vmdir/
The tool generates files in /var/log/vmware/vmdir labeled "Integrity---" for each node in ELM.
The contents of the file will be objects that differ or are missing between the nodes. i.e. "I" at the start of DN represents the entry is inconsistent, "M" at the start of DN represents entry is not present.
Running the tool twice with 5 minutes in between would allow you to compare the entries that are inconsistent -- if the inconsistent items are the same in both executions, then you need to resolve the inconsistencies.
If inconsistencies are found:

Follow the steps in Fixing VMDIR inconsistencies with SSO domain repoints