Addon is not updated in guest cluster even through repository has been updated in Supervisor

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Addon repository is updated in Supervisor by following Update an Add-on Repository Install
• Packages and apps in guest cluster are not updated. For example, the images of these resources are still referencing the old repository. 
• The PackageRepository (pkgr) in guest cluster is failed to reconcile:
  
  # kubectl get pkgr -A
NAMESPACE           NAME                AGE   DESCRIPTION                                                                                PAUSED
vmware-system-tkg   standard-packages   97d   Reconcile failed: Fetching resources: Error (see .status.usefulErrorMessage for details)

# kubectl describe pkgr -n vmware-system-tkg   standard-packages
...
  usefulErrorMessage: |
    vendir: Error: Syncing directory '0':
      Syncing directory '.' with imgpkgBundle contents:
        Fetching image:
          Error while preparing a transport to talk with the registry:
            Unable to create round tripper:
              Get "https://<registry-fqdn>/v2/": dial tcp <registry-ip-address>:443: i/o timeout

Environment

VMware vSphere Kubernetes Service

Cause

Guest cluster can not connect to the registry that causes the addon failed to be updated in guest cluster. 

Resolution

The issue is caused by the network connection between guest cluster and registry. Follow the steps to identify the networking issue:

1. SSH to the guest cluster control plane node by following the steps in Accessing vSphere with Tanzu workload clusters using SSH
2. nslookup the FQDN of registry to verify if the DNS resolution is correct or not.
3. Ping the IP address or FQDN of registry.
4. Ping the gateway of workload network.

After identify the problematic networking, engage networking team to check the network components, such as firewall, router, physical switch. 

Additional Information

Troubleshooting virtual machine network connection issues