Using vmdir_tool.py to identify vmdir/ELM replication inconsistencies
search cancel

Using vmdir_tool.py to identify vmdir/ELM replication inconsistencies

book

Article ID: 367525

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

VMDIR Tool is a tool used to address issues with data stored in the ELM database.  The tool can be used to detect and correct problems that could cause failures in topology changes (converge, repoint, etc.), upgrades, or failures incurred as a result of maintenance (e.g. incorrectly applying new SSL certificates).  This article will outline its functions and use.

  • VMdird replication issues.
  • You find error messages in the vmdird-syslog.log (or vmdird.log) file located in /var/log/vmware/vmdird/ 
    err vmdird t@139841484338944: _VmDirSearchPreCondition: Server in not in normal mode, not allowing outward replication.
    err vmdird  t@140609914369792: UpdateServerObject: InternalModifyEntry failed. Error code: 53, Error string: Server in read-only mode
  • When querying the vmdir state using vdcadmintool, you get a 'Null' or 'Read Only' state
  • Querying replication status shows 'Status available: No" 
  • Other symptoms, such as: 
    • User/Group changes are not propagating to other vCenters in the SSO domain
    • Cannot repoint, upgrade, converge
    • Cannot register other vSphere products

Environment

Available in vCenter server 7.0 and above.

Cause

When vCenter servers in ELM are restored from snapshots improperly, inconsistencies in the VMDIR database are introduced.  The database will attempt to patch any holes, but this may not always be successful.  This tool can help identify and troubleshoot such scenarios.

For details on safe snapshots, see  VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

Resolution

Tool Use

This tool should primarily be used to identify ELM database inconsistencies.  NOTE:  Only the read functions integrityCheck, replicationValidation, and getTopology options should be used.  Other options may be destructive.

 

Tool directory:

/usr/lib/vmware-vmdir/vmdir-tool/

 

View the help menu:

# python /usr/lib/vmware-vmdir/vmdir-tool/vmdir_tool.py --help
usage: vmdir_tool.py [-h] [--hostname HOSTNAME] [--getTopology] [--integrityCheck] [--hostOnly] [--replicationValidation]
                     [--uptodateVectorCheck] [--restoreVmdir] [--machineAccount]

VMDir Debug Tool

options:
  -h, --help            show this help message and exit
  --hostname HOSTNAME   Hostname or FQDN of the node
  --getTopology         Store Topology information of the nodes in /var/log/vmware/vmdir/topology_info.json
  --integrityCheck      Check data inconsistency between all nodes. Generated logs can be found in /var/log/vmware/vmdir. NOTE:The output
                        of this check is not guranteed to be complete always in a log support bundle. In such cases it must be run
                        separately to collect the complete output.
  --hostOnly            Used with integrityCheck flag to perform checks only from the specified node for hostname
  --replicationValidation
                        Validate replication between all the nodes
  --uptodateVectorCheck
                        Validate consistency between vmdir data & up-to-date vector
  --restoreVmdir        Restore Vmdir
  --machineAccount

 

Initiate an integrity check to compare database entries between nodes:

python /usr/lib/vmware-vmdir/vmdir-tool/vmdir_tool.py --hostname $HOSTNAME --integrityCheck --machineAccount

NOTE: The '$HOSTNAME' value will be automatically expanded to the current vCenter hostname.  The value will not need to change.  Additionally, the --machineAccount parameter will run the tool without having to provide a username and password.

 

Tool results:

All tool results and logs are in  the directory: /var/log/vmware/vmdir/.  The tool generates files in /var/log/vmware/vmdir labeled "Integrity-<vc1hostname>-<vc2hostname>-<date>" for each node in ELM.  The contents of the file will be objects that differ or are missing between the nodes.  i.e. "I" at the start of DN represents the entry is inconsistent, "M" at the start of DN represents entry is not present.  Running the tool twice with 5 minutes in between would allow you to compare the entries that are inconsistent -- if the inconsistent items are the same in both executions, then you need to resolve the inconsistencies (see below).

 

If inconsistencies are found:

please follow the steps in Fixing VMDIR inconsistencies with SSO domain repoints

 

 

 

Powered by Wolken Software