vCenter Content Library sync fails with error "Reached the maximum number of retries (10)"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Subscribed libraries might fail to sync changes on items
• Messages similar to the following might be seen in /var/log/vmware/content-library/cls.log.

error = TransferError (com.vmware.transfer.transfer_error) => {
        message = Reached the maximum number of retries (10) while attempting to transfer from com.vmware.transfer.impl.HttpClientEndpointImpl@c123456. Failing transfer.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x

Cause

 

This error is typically caused by thread exhaustion in either the Publisher or Subscriber library. Thread exhaustion happens when too many items attempt to sync concurrently between Content Libraries, reaching one of the limits outlined in "How to update Content Library Settings." In essence, a library item starts to sync, but not all of its files can secure an open thread, leading to retries. If a file fails to secure a thread within 10 retries, it results in a sync task failure. Usually, a manual sync of the item afterwards will successfully transfer the missing files, provided the Content Library is no longer overwhelmed and has available threads.

 

 

Originally, this article addressed a different (less common) issue where multiple changes to content library items could be prioritized and executed out of order on the subscribed vCenter during a sync operation, causing conflicts. For instance, if an item is renamed and then deleted, the subscriber side might receive these changes out of order, attempting to delete it before renaming it, resulting in a sync failure. Although this issue is still being addressed, it is less common than thread exhaustion.

 

In this situation, you might see a correlated log entry like the following:

  defaultMessage = Operation is not supported since a library item with the name xdt_isp exists in the library 'TestSubscriber' (ID: xxxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxx).,
  args = [xdt_isp, CLS-Subscriber-library, xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx],

  errorType = ALREADY_EXISTS

Resolution

To alleviate thread exhaustion during sync operations, consider the following measures:

• Limit the number of subscriber libraries per publisher.
• Stagger auto-sync start and stop times for subscriber libraries connected to the same publisher.
• Disable auto-sync for subscribing content libraries, enabling manual syncing as needed.
• Reduce the number of items a subscriber library attempts to sync simultaneously.
• Increase the thread pools for content library transfer sessions. This may require trial and error, as each environment and its library items can vary significantly.

Please see "How to update Content Library Settings" for details on the settings described above

 

To workaround the issue where content library syncs items in the incorrect order, issue a manual sync on an item before making any further changes that might conflict with a previous change if read out of order.

Additional Information

A restart the services, vmware-content-library, might be needed following the changes. To restart the service, please see the following KB: https://knowledge.broadcom.com/external/article?articleNumber=344633