Error "error: while downloading images: Retried 5 times. read tcp ########### -> ###########:443: read: connection reset by peer." while downloading large size image bundle during upgrade

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• Downloading large size TKG image bundle during upgrade using the below command does not complete successfully.
  
  tanzu isolated-cluster download-bundle --source-repo <SOURCE-REGISTRY> --tkg-version <TKG-VERSION> --ca-certificate <SECURITY-CERTIFICATE>

Environment

TKG 2.5.4

Cause

• The "read: connection reset by peer" error during the tanzu isolated-cluster download-bundle execution indicates that an intermediary network device is forcibly closing the TCP connection.
• If the failure consistently occurs at a specific size limit, it is likely that a firewall, proxy, or VPN is enforcing a session timeout or size limit.
• The Tanzu CLI’s underlying imgpkg tool attempts five retries after these network resets terminate the socket and the download fails.

Resolution

• Transition to a list-and-pull methodology to avoid payload size and timeout constraints.
• Instead of using the native download-bundle command, which creates a single large file prone to resets, we are generating the artifact manifest and looping retrieval through individual imgpkg processes.
• This approach, avoids any size limit and timeout constraints with network.

Please find the step by step instructions below

1. Validate available block storage on the target partition and ensure there is sufficient space.
   
   df -h 
   
   
2. Generate the image and bundle lists
   
   tanzu isolated-cluster list-bundle  --source-repo projects.registry.vmware.com/tkg --tkg-version v2.5.4
   
   
3. Confirm the presence of the output configuration files
   
   ls -l bundle_list_v2.5.4.conf image_list_v2.5.4.conf

4. Update the local retrieval process to use segmented downloads via imgpkg instead of a single stream. By wrapping the tool in a loop to download images and bundles iteratively, the script now includes a check to safely resume execution following network interruptions.
   
   

    Define target staging directory with sufficient block storage
   STAGING_DIR="/mnt/tkg-offline"
   mkdir -p "${STAGING_DIR}/bundles" "${STAGING_DIR}/images"
   
   echo "Executing segmented bundle extraction..."
   while read -r bundle; do
     # Generate a safe filesystem name from the registry path
     safe_name=$(echo "$bundle" | awk -F'/' '{print $NF}' | tr ':' '-')
     target_tar="${STAGING_DIR}/bundles/${safe_name}.tar"
     
     if [ ! -f "$target_tar" ]; then
       echo "Downloading Bundle: ${bundle}"
       imgpkg copy -b "${bundle}" --to-tar "$target_tar"
     else
       echo "Skipping Bundle: ${bundle} - Artifact verified on disk"
     fi
   done < bundle_list_v2.5.4.conf
   
   echo "Executing segmented image extraction..."
   while read -r image; do
     safe_name=$(echo "$image" | awk -F'/' '{print $NF}' | tr ':' '-')
     target_tar="${STAGING_DIR}/images/${safe_name}.tar"
     
     if [ ! -f "$target_tar" ]; then
       echo "Downloading Image: ${image}"
       imgpkg copy -i "${image}" --to-tar "$target_tar"
     else
       echo "Skipping Image: ${image} - Artifact verified on disk"
     fi
   done < image_list_v2.5.4.conf

5. Publish the bundles and images file to the target offline registry using imgpkg and ensure the original repository path structure is maintained to comply with TKr resolution rules
   
   

   # Define the target air-gapped registry FQDN and target path
   
   OFFLINE_REGISTRY="<REDACTED_HOSTNAMES>/tkg"
   
   echo "Ingesting discrete bundles into offline registry..."
   for tar in /mnt/tkg-offline/bundles/*.tar; do
     imgpkg copy --tar "$tar" --to-repo "${OFFLINE_REGISTRY}"
   done
   
   echo "Ingesting discrete images into offline registry..."
   for tar in /mnt/tkg-offline/images/*.tar; do
     imgpkg copy --tar "$tar" --to-repo "${OFFLINE_REGISTRY}"
   done

6. Validate the packages after uploading to private registry using native Carvel packaging utility (imgpkg), which is bundled with the Tanzu CLI dependencies, to query registry tags
   
   
   • Execute the imgpkg tag list command against your specific Harbor FQDN and project path: imgpkg tag list -i <HARBOR_FQDN>/<PROJECT_NAME>/tkg-compatibility
   • The returned list must explicitly contain the tag matching your target Tanzu management plugin version (e.g., v0.32.5).
   • The execution standard output generates a structured list containing the repository name and all associated tags.
   • This confirms the structural availability of the requisite binaries within the registry tier prior to initiating state-changing operations like cluster upgrades.

Additional Information

Prepare an Internet-Restricted Environment - Broadcom TechDocs