Using the lw-measure tool to detect latency between vCenter Server and Active Directory domain controllers

Products

• VMware vCenter Server

Issue/Introduction

This article shows how to use the lw-measure tool to detect latency between vCenter Server and Active Directory Domain controllers.

Symptoms:
Slow or failed authentication due to vCenter/domain controller latency.

Resolution

Using the lw-measure tool

This tool should be run from a PSC or embedded vCenter Server. It will determine the network latency while connecting to the Domain Controllers in a domain as well as the time taken for user lookups of that domain.

For vCenter Server 7.0 and above, skip to step 5 as lw-measure is included in /opt/likewise/bin/lw-measure.

1. Download the lw_measure binary attached to this article.
2. From an SSH session to the vCenter Appliance, create a new directory for the tool

mkdir -p /root/debugging
cd /root/debugging

3. Move the lw_measure file to the /root/debugging directory
4. Make the file executable

chmod +x /root/debugging/lw-measure

5. Run lw-measure for 600 seconds (10 mins) from the directory its located in (/opt/likewise/bin/ in 7.0)

nohup ./lw-measure <Domain Name> <AD user UPN> <Time in Seconds> 2>&1 &

Example: nohup ./lw-measure example.com [email protected] 600 2>&1 &

6. Use ctrl-c to return to the shell. This does not terminate the process, which will continue for the number of seconds specified in the command.

The output is saved to a file in the current working directory named nohup.out

The output will look like this:
Passed Parameters: example.com, [email protected], 600
Time(usec) | DC_Index | DC_Name | DC_Address | User_Sid | User_Info | GetUserSid_Error | GetUserInfo_Error | Call_Time_GetUserSid(usec) | Call_Time_GetUserInfo(usec)
 
Found DC at: 12345678914490 | Domain: example.com | Number of DCs: 2
1571752685655671 | 0 | example-dc1.example.com | 10.10.10.1 | S-1-5-21-1234567890-1234567890-1234567890-500 | CN=Administrator,CN=Users,DC=example,DC=com | 0 | 0 | 5607301 | 1511252
1571752692701943 | 1 | example-dc2.example.com | 10.10.10.2 | S-1-5-21-1234567890-1234567890-1234567890-500 | CN=Administrator,CN=Users,DC=example,DC=com | 0 | 0 | 5557150 | 1488192
Found DC at: 12345678914490 | Domain: example.com | Number of DCs: 2
1571752700338333 | 0 | example-dc1.example.com | 10.10.10.1 | S-1-5-21-1234567890-1234567890-1234567890-500 | CN=Administrator,CN=Users,DC=example,DC=com | 0 | 0 | 4932434 | 1491279
1571752706762249 | 1 | example-dc2.example.com | 10.10.10.2 | S-1-5-21-1234567890-1234567890-1234567890-500 | CN=Administrator,CN=Users,DC=example,DC=com | 0 | 0 | 4933276 | 1489925

Observe the output. If either of the last two columns has latencies more than 5,000,000 usec (5 seconds), it indicates a significant delay in the communication between vCenter and the domain controller. Even authentications with latencies below 5 seconds can cause problems if there are too many of them for likewise to keep up with.


Caution: Using an incorrect value for the domain name can leave a stale process of lw_measure after it is exited. See the example for the proper format. Use ps auxxx| grep lw-measure to find and kill these processes if they exist.

Additional Information

For information on using checkADConfig to detect connectivity and DNS issues between vCenter Server and Active Directory, see https://kb.vmware.com/s/article/79323