Kapp-controller is failing to pull imgpkg for certain supervisor services

Products

• VMware vSphere with Tanzu

Issue/Introduction

Kapp-controller is failing to pull imgpkg images with the following error:

status:
  conditions:
  - message: 'Fetching resources: Error (see .status.usefulErrorMessage for details)'
    status: "True"
    type: ReconcileFailed
  consecutiveReconcileFailures: 22
  fetch:
    error: 'Fetching resources: Error (see .status.usefulErrorMessage for details)'
    exitCode: 1
    startedAt: "2024-10-11T13:57:42Z"
    stderr: |
      vendir: Error: Syncing directory '0':
        Syncing directory '.' with imgpkgBundle contents:
          Fetching image:
            Error while preparing a transport to talk with the registry:
              Unable to create round tripper:
                Get "https://projects.packages.broadcom.com/v2/": proxyconnect tcp: dial TCP <PROXYIP::PROXYPORT>: i/o timeout

Note: In this instance, a proxy was configured.

Environment

This was observed in vCenter 8.0U3. However, the KAPP-controller that performed the Supervisor Services install defaults to use the management network (eth0) to pull imgpkg bundles.

Note: In this instance, a proxy was configured.

Cause

Kapp-controller traffic will be sent through the physical interface for the Management Network on the Supervisor Control Plane VM.
Firstly, you will need SSH access to the supervisor control plane VMs. See the Section "How to SSH into Supervisor Control Plane VMs" from the KB  Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's.

This can be observed by exec'ing into the kapp-controller container or pod and running tcpdump -i eth0 host <address to ping>

1) Exec into kapp-controller pod. See Get a Shell to a Running Container

kubectl exec --stdin --tty kapp-controller-<uniquestring> -- /bin/bash

2) Start tcpdump in another shell on the Supervisor Control Plane VM where the kapp-controller is running. You can find this using the -o wide option for kubectl get pods.

To find the Supervisor Control Plane Node where kapp-controller pods are running.
kubectl get pods -A -o wide | grep kapp

To start a packet capture on the management interface and filter for a specific IP. In this instance, we used the Proxy IP. You may need to query for the image package URL directly.
tcpdump -i eth0 host <proxy IP>

3) Begin pinging proxy IP from kapp-controller shell

We can see that ICMP traffic is passing over eth0 of the management network of the Supervisor Control Plane VM. When we change the tcpdump capture interface to eth1 we do not see this ICMP traffic.

Resolution

Kapp-controller is using the management network interface of the Supervisor Control Plane VM to pull carvel imgpkgs.