Pulling Tanzu packages from projects.registry.vmware.com fails to pull image for pods showing read: connection reset by peer

Products

• VMware vSphere with Tanzu
• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Kubernetes Grid Service (TKGs)
• Tanzu Kubernetes Grid

Issue/Introduction

When you describe a failing pod you see similar to the following

Warning  Failed     38s                kubelet            Failed to pull image "projects.registry.vmware.com/tkg/fluent-bit@sha256:64685**************************************": rpc error: code = Unknown desc = failed to pull and unpack image "projects.registry.vmware.com/tkg/fluent-bit@sha256:64685******************************": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-broadcom/filestore/45/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx read: connection reset by peer

Example above is taken as similar representation of the error shown when a failing pod is described

Environment

• 7.0.x
• 8.0.x

Cause

The issue is caused by failure to access the Tanzu package repositories.

This issue for Tanzu environments occurs where controls are in place to apply restrictions on egress from the environment.

This issue can happen as well due to to the following redirect from “projects.registry.vmware.com” -> “projects.packages.broadcom.com” -> “jfrog-prod-usw2-shared-oregon-main.s3", this is why “jfrog-prod-usw2-shared-oregon-main.s3" has to be whitelisted. 

Resolution

The following domains should be whitelisting on firewall controlled environments if you want to download packages from the Tanzu repositories.

wp-content.vmware.com
*.tmc.cloud.vmware.com
projects.registry.vmware.com
projects.packages.broadcom.com
jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com