Content library sync fails with error "IO error while uploading to datastore"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Sync may fail for one or more templates
• Manually importing the affected template to the subscriber vCenter content library may also fail
• Manual import process reaches 100% in the UI and errors out after 2 minutes
• Templates contain large thin provisioned disks (2 TB+)
• Smaller templates may fail if there are any performance bottlenecks in the environment.
• Subscriber vCenter - /var/log/vmware/content-library/cls.log

YYYY-MM-DDTHH:MM:SSZ | ERROR    | null             | transferService-nfc-#####-test.vmdk-upload | NfcEndpointImpl                | Session ###: IO error while uploading to datastore (ds:///vmfs/volumes/###//contentlib-###/###/test.vmdk?serverId=###): Read timed out
  
  
• /var/run/log/hostd.log on the ESXi host participating in the upload

YYYY-MM-DDTHH:MM:SSZ warning hostd[2100826] [Originator@6876 sub=Libs opID=###] [NFC ERROR]NfcNetTcpSetError: Connection reset by peer
YYYY-MM-DDTHH:MM:SSZ warning hostd[2100826] [Originator@6876 sub=Libs opID=###] [NFC ERROR]NfcNetTcpRead: bRead: -1. errno: 104.
YYYY-MM-DDTHH:MM:SSZ warning hostd[2100826] [Originator@6876 sub=Libs opID=###] [NFC ERROR]NfcNet_Recv: requested 264, recevied only 0 bytes
YYYY-MM-DDTHH:MM:SSZ warning hostd[2100826] [Originator@6876 sub=Libs opID=###] [NFC ERROR]NfcGetMessage: NfcNet_Recv failed: NFC_NETWORK_ERROR
YYYY-MM-DDTHH:MM:SSZ warning hostd[2100826] [Originator@6876 sub=Libs opID=###] [NFC ERROR]NfcServerProcessClientMsg: Failed to get message from client: NFC_NETWORK_ERROR (The operation experienced a network error)
YYYY-MM-DDTHH:MM:SSZ error hostd[2100826] [Originator@6876 sub=Nfcsvc opID=###] Read error from the nfcLib: NFC_NETWORK_ERROR (done = yep)


Environment

• vCenter Server 7.x
• vCenter Server 8.x

Cause

Content library has a two-minute timeout set for the NFC transfer session. For larger disks, at the end of the data transfer, it may take more than two minutes to complete the NFC transfer. This causes the manual import or the sync to fail.

Resolution

Workaround

Increase the timeout values.

1. SSH to the Subscriber vCenter
   
   
2. Navigate to the content library configuration directory
   
   cd /etc/vmware-content-library/config/
   
   
3. Backup the original file
   
   cp ts-config.properties ts-config.properties.orig
   
   
4. Edit the file
   
   vi ts-config.properties
   
   
5. Modify the timeout value of below two parameters from 120 to 240
   
   transfer.nfcClient.socket.timeout.seconds=240
transfer.http.request.timeout.seconds=240
   
   
6. Save the file
   
   
7. Restart the content library service
   
   service-control --stop vmware-content-library
service-control --start vmware-content-library
   
   
8. Sync the affected template