Kapp-controller is failing to pull imgpkg images with the following error:
status:
conditions:
- message: 'Fetching resources: Error (see .status.usefulErrorMessage for details)'
status: "True"
type: ReconcileFailed
consecutiveReconcileFailures: 22
fetch:
error: 'Fetching resources: Error (see .status.usefulErrorMessage for details)'
exitCode: 1
startedAt: "2024-10-11T13:57:42Z"
stderr: |
vendir: Error: Syncing directory '0':
Syncing directory '.' with imgpkgBundle contents:
Fetching image:
Error while preparing a transport to talk with the registry:
Unable to create round tripper:
Get "https://projects.packages.broadcom.com/v2/": proxyconnect tcp: dial TCP <PROXYIP::PROXYPORT>: i/o timeout
Note: In this instance, a proxy was configured.
This was observed in vCenter 8.0U3. However, the KAPP-controller that performed the Supervisor Services install defaults to use the management network (eth0) to pull imgpkg bundles.
Note: In this instance, a proxy was configured.
Kapp-controller traffic will be sent through the physical interface for the Management Network on the Supervisor Control Plane VM.
Firstly, you will need SSH access to the supervisor control plane VMs. See the Section "How to SSH into Supervisor Control Plane VMs" from the KB Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VM's.
This can be observed by exec'ing into the kapp-controller container or pod and running tcpdump -i eth0 host <address to ping>
1) Exec into kapp-controller pod. See Get a Shell to a Running Container
kubectl exec --stdin --tty kapp-controller-<uniquestring> -- /bin/bash
2) Start tcpdump in another shell on the Supervisor Control Plane VM where the kapp-controller is running. You can find this using the -o wide option for kubectl get pods.
To find the Supervisor Control Plane Node where kapp-controller pods are running.
kubectl get pods -A -o wide | grep kapp
To start a packet capture on the management interface and filter for a specific IP. In this instance, we used the Proxy IP. You may need to query for the image package URL directly.
tcpdump -i eth0 host <proxy IP>
3) Begin pinging proxy IP from kapp-controller shell
We can see that ICMP traffic is passing over eth0 of the management network of the Supervisor Control Plane VM. When we change the tcpdump capture interface to eth1 we do not see this ICMP traffic.
Kapp-controller is using the management network interface of the Supervisor Control Plane VM to pull carvel imgpkgs.