Supervisor PKGI ReconcileFailed - certificate is valid for ingress.local, not packages.broadcom.com

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

One or more packageInstalls (PKGI) in the Supervisor cluster are in ReconcileFailed state.

kubectl get pkgi -A

When describing the ReconcileFailed PKGI, an error message similar to the following is displayed, where the packages URL noted can vary by each PKGI:

Useful Error Message:  vendir: Error: Syncing directory '0';
Syncing directory '.' with imgpkgBundle contnets:
   Fetching image:
      Error while preparing a transport to talk with the registry:
             Unable to create round tripper:
                        Get "https://projects.packages.broadcom.com/v2/": tls: failed to verify certificate: x509: certificate is valid for ingress.local, not projects.packages.broadcom.com

Environment

vSphere Supervisor

VMware Cloud Foundation Automation

Cause

A DNS networking change or new DNS entry is intercepting the connection to the Broadcom repository and inserting the noted "ingress.local" certificate.

Kapp-controller pulls images for packageInstalls (PKGI) and uses ndots:5 to resolve DNS.

Resolution

Look into any recent DNS changes in the environment.

More details on the intercepting connection can be found from checking with the kapp-controller on the Supervisor cluster:

1. Connect into the Supervisor cluster
   
   
2. Locate the kapp-controller pod:
   

   kubectl get pods -n vmware-system-appplatform-operator-system | grep kapp

    

3. Exec into the kapp-controller pod with bash shell:
   

   kubectl exec -it -c kapp-controller -n vmware-system-appplatform-operator-system <kapp-controller-pod> -- /bin/bash

    

4. In the kapp-controller bash shell session, perform a curl command to the Broadcom repository from the error message over eth0:
   

   kapp-controller [ / ]$ curl -v https://projects.packages.broadcom.com/v2/:443 --interface eth0
   
   * Host projects.packages.broadcom.com:443 was resolved.
   * IPv6: (none)
   * IPv4: <intercepting private IP address>
   *   Trying <intercepting private IP address>:443...

   Look into the IP address noted in the output and any networking changes regarding this IP address. It could be a DNS server problem, search domain issue or a DNS entry.
   
   This IP address is intercepting the connection to the Broadcom repository and interjecting with the "ingress.local" certificate.