Unable to sync catalog to some subscriber Cloud Director sites while others are able to sync from the provider

Products

• VMware Cloud Director

Issue/Introduction

Some Cloud Director sites are unable to fully sync a catalog from a Cloud Director provider.
Some catalog items successfully sync, while others may fail.
Cloud Director sites which are using a WAN internal route to a single cell on the Provider successfully sync the entire catalog consistently.
Cloud Director sites traversing the internet via a load balancer may have some catalog items sync, and others fail to sync.

Environment

Cloud Director 10.6.0.1

Cause

vCloud Container Debug log shows an error similar to:

 

2025-03-14 17:38:29,542 | WARN     | worker-urn:uuid:########-####-####-####-############-########## | TransferEndpointImpl           | Received exception Read timed out for offset 1,744,830,464 while storing file /example-vmdk-name.vmdk. Retrying |
2025-03-14 17:38:29,548 | WARN     | worker-urn:uuid:########-####-####-####-############-########## | TransferEndpointImpl           | Received exception Unexpected end of ZLIB input stream for offset 1,744,830,464 while storing file /example-vmdk-name.vmdk. Retrying |                                      
2025-03-14 17:38:29,551 | WARN     | worker-urn:uuid:########-####-####-####-############-########## | TransferEndpointImpl           | Giving up for /example-vmdk-name.vmdk offset 1,744,830,464 after attempt 3. Error java.io.EOFException: Unexpected end of ZLIB input stream |                    
2025-03-14 17:38:29,551 | WARN     | worker-urn:uuid:########-####-####-####-############-########## | TransferEndpointImpl           | Error transferring file from https://<provider-public-url>/vcsp/lib/example-vmdk-name.vmdktoVfsFile[fileObject=file:///example-vmdk-name.vmdk] |                                                            java.io.EOFException: Unexpected end of ZLIB input stream                                                    
       at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
       at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
       at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
       at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:70)
       at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1560)
       at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1518)
       at com.vmware.vcloud.common.vfs.chunked.VfsChunkedContent.addChunk(VfsChunkedContent.java:93)
       at com.vmware.ts.impl.TransferEndpointImpl.storeFileInChunks(TransferEndpointImpl.java:977)
       at com.vmware.ts.impl.TransferEndpointImpl.attemptDownload(TransferEndpointImpl.java:887)
       at com.vmware.ts.impl.TransferEndpointImpl$VfsChunkDownloadWorker.call(TransferEndpointImpl.java:728)
       at com.vmware.ts.impl.TransferEndpointImpl$VfsChunkDownloadWorker.call(TransferEndpointImpl.java:704)
       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       at com.vmware.vcloud.common.threading.NDCContextAwareThreadPoolExecutor$NDCRunnableFutureAdapter.run(NDCContextAwareThreadPoolExecutor.java:77)
       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       at java.base/java.lang.Thread.run(Thread.java:829) 

Resolution

Check to make sure the load balancer is configured as recommended in VMware Cloud Director Cell Load Balancing Configuration.