Upgrade of TKGS fails with ReconcileFailed

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

1. Symptom in runtime-extension package:

   “Reason: ReconcileFailed. Message: kapp: Error: waiting on reconcile packageinstall/runtime-extension (packaging.carvel.dev/v1alpha1) namespace: svc-tkg-domain-c8: Finished unsuccessfully (Reconcile failed: (message: kapp: Error: create extensionconfig/runtime-extension (runtime.cluster.x-k8s.io/v1alpha1) cluster: Creating resource extensionconfig/runtime-extension (runtime.cluster.x-k8s.io/v1alpha1) cluster: API server says: admission webhook "validation.extensionconfig.runtime.cluster.#-k8s.io" denied the request: spec: Forbidden: can be set only if the RuntimeSDK feature flag is enabled (reason: Forbidden))).”

    

2. The capi-controller-manager-########dc-#### pods have the message:

   Failed to pull image "projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.2.0/tkg-service@sha256:#############################": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.2.0/tkg-service@sha256:#############################": failed to resolve reference "projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.2.0/tkg-service@sha256:#############################": failed to do request: Head "https://projects.packages.broadcom.com/v2/vsphere/iaas/tkg-service/3.2.0/tkg-service/manifests/sha256:#############################": dial tcp ###.##.###.##:443: i/o timeout

3. Describe of the tkg-service pkgi has error on capw deployment:

   Reconcile failed:  (message: kapp: Error: waiting on reconcile deployment/capw-controller-manager (apps/v1) namespace: svc-tkg-domain-cXX)

    

4. CAPW pod is in ImagePullBackOff state with describe of the capw pod showing below failure:

   Warning  Failed   51m (x6129 over 8d)   kubelet  (combined from similar events): Failed to pull image "projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.2.0/tkg-service@sha256:#############################": failed to pull and unpack image "projects.packages.broadcom.com/vsphere/iaas/tkg-service/3.2.0/tkg-service@sha256:#############################": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-broadcom/filestore/b1...

Environment

vSphere with Tanzu

Cause

• The inability of CAPI and CAPW deployments to pull images is caused by restricted access to Tanzu package repositories.
• If port 443 is not open to "s3-us-west-2-w.amazonaws.com" and "projects.packages.broadcom.com," the system encounters a 'Forbidden' error.

Resolution

• Open port 443 access to "s3-us-west-2-w.amazonaws.com" & "projects.packages.broadcom.com" websites.  
  After the access has been opened packages might be in a pause state, follow kb: Mitigate error condition 'admission webhook "admission.vmware.com" denied the request'
  
  
• The following domains should be whitelisting on firewall controlled environments if you want to download packages from the Tanzu repositories.
  • wp-content.vmware.com
  • *.tmc.cloud.vmware.com
  • projects.registry.vmware.com
  • projects.packages.broadcom.com
  • jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com
    
    Refer Pulling Tanzu packages from projects.registry.vmware.com fails to pull image for pods showing read: connection reset by peer for details.