Pulling Tanzu packages from projects.registry.vmware.com fails to pull image for pods with error read: connection reset by peer

Products

• VMware vSphere Kubernetes Service
• VMware Tanzu Kubernetes Grid Integrated (TKGi)
• VMware Tanzu Kubernetes Grid Service (TKGs)
• Tanzu Kubernetes Grid

Issue/Introduction

Upon describing a pod in failed state, following error is seen : 

kubectl describe pod <pod_name> -n <pod_namespace> 

Warning  Failed     38s                kubelet            Failed to pull image "projects.registry.vmware.com/tkg/fluent-bit@sha256:64685**************************************": rpc error: code = Unknown desc = failed to pull and unpack image "projects.registry.vmware.com/tkg/fluent-bit@sha256:64685******************************": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-broadcom/filestore/45/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx read: connection reset by peer

Environment

VMware vSphere Kubernetes Service

Cause

• The issue is caused by failure to access the Tanzu package repositories.
• This issue for Tanzu environments occurs where controls are in place to apply restrictions on egress from the environment.
• This issue can happen due to the following redirect from “projects.registry.vmware.com” -> “projects.packages.broadcom.com” -> “jfrog-prod-usw2-shared-oregon-main.s3".

Resolution

The following domains should be whitelisted on firewall controlled environments to download packages from the Tanzu repositories:

• wp-content.vmware.com
• *.tmc.cloud.vmware.com
• projects.registry.vmware.com
• projects.packages.broadcom.com
• jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com
• s3-us-west-2-w.amazonaws.com