REPO_SYNC remains "IN PROGRESS" for an abnormally long time

Products

• VMware NSX

Issue/Introduction

• Under "View Details", REPO_SYNC remains "IN PROGRESS" for a long time, longer than expected.
  For example, more than one hour.
• Observing the NSX API logs, HTTPS transactions for even small files (such as .sh or .txt) take a long time.
  For example, more than 20 seconds for a file that is only a few KB. You can check the file size on the source NSX Manager with the path from the log line: ls -lh /repository/####
  Log lines similar to the below are encountered on the NSX Manager in /var/log/proton/nsxapi.log

  RepoSyncThread-## DefaultCommandExecutionTask ## SYSTEM [nsx@## comp="nsx-manager" level="INFO" subcomp="manager"] Time taken to execute Command CommandImpl [getCommandList()=[/usr/bin/python3, /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper, --silent, --head, https://####:443/repository/####, --show-error, --thumbprint, ####], getName()=null, getExecutionDirectory()=/tmp, getCommandExecutionConfiguration()=CommandExecutionConfigurationImpl [redirectErrorStream=true, timeout=0, properties={}]] is ## seconds

• Manually running the command from the above log line shows that the query takes longer than expected:
  (Copy the command contained within the array and remove the commas.)

  time /usr/bin/python3 /opt/vmware/nsx-common/python/nsx_utils/curl_wrapper --silent --head https://####:443/repository/#### --show-error --thumbprint ####
  [...]
  real    #m##.##s

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

A delay is introduced by the underlying network for each individual file being synced, either during resolution of the remote NSX Manager FQDN or during the actual HTTPS transaction.
Because REPO_SYNC will synchronise a large number of files between NSX Manager appliances, if every transaction take longer than expected, the overall completion time can be considerable.

• You can validate the DNS resolution time of the remote NSX Manager FQDN:

  time dig +short ####
  real    #m##.##s

• You can validate the HTTPS connection time, using the remote NSX Manager FQDN, then comparing with using the remote NSX Manager IP address:

  time curl -sk #### > /dev/null
  real    #m##.##s

Resolution

This is a condition that may occur in a VMware NSX environment.

To resolve this issue, address the underlying cause that introduces delay in the HTTPS transactions performed by REPO_SYNC.