Supervisor Cluster Configuring with Supervisor Services ReconcileFailed DNS lookup failure to projects.packages.broadcom.com

Products

VMware vSphere Kubernetes Service

Issue/Introduction

The affected Supervisor cluster is stuck in Configuring state with an error message similar to the below:

Configured Core Supervisor Services
Service: tkg.vsphere.vmware.com. Reason: ReconcileFailed. Message: vendir: Error: Syncing directory '0': Syncing directory '.' with imgpkgBundle contents: Fetching image: Error while preparing a transport to talk with the registry: Unable to create round tripper: Get "https://projects.packages.broadcom.com/v2/": dial tcp: lookup projects.packages.broadcom.com on 127.0.0.53:53: read udp 127.0.0.1:60147->127.0.0.53:53: i/o timeout .

In the Supervisor cluster environment, the following symptoms are observed:

Supervisor services are configured to pull their images from the projects.packages.broadcom.com repository and not from a private registry
Viewing the status of the Kubernetes Service supervisor service in the vSphere web UI shows that its Signature is Unknown or Untrusted
- This is the equivalent Supervisor service for the tkg.vsphere.vmware.com packageInstall (pkgi)
From each Supervisor VM, you are able to successfully curl the IP for the Broadcom public repository, but not its alias "projects.packages.broadcom.com"
- If the VM is able to reach the IP, this indicates a DNS issue rather than a connection issue
- This error can appear if one or more Supervisor VMs cannot reach the Broadcom public repository by its FQDN
In the Supervisor cluster's context, one or more supervisor services are in ReconcileFailed state with the above lookup error:
```
kubectl get pkgi -A
```
While SSH directly to each Supervisor VM, at least one of the VMs has the following latest messages in the systemd-resolved journal logs, showing incorrect DNS server IP(s):
```
journalctl -xeu systemd-resolved

Using degraded feature set UDP instead of TCP for DNS server <incorrect IP>.
Using degraded feature set TCP instead of UDP for DNS server <incorrect IP>.
Using degraded feature set UDP instead of UDP+EDNS0 for DNS server <incorrect IP>.
```
This can also occur if the DNS server IPs do not match the DNS servers configured in the Supervisor cluster's Management Network or Workload Network.
The Supervisor VM with the above issues is also unable to reach the vCenter VM's FQDN.

This issue can occur during or after a Supervisor cluster upgrade.

Environment

vSphere Supervisor 8.x

vSphere Supervisor 9.x

Supervisor services are configured to pull their images from the projects.packages.broadcom.com repository and not from a private registry

Cause

The systemctl service serving DNS on the Supervisor cluster is using the incorrect configuration for the Supervisor cluster's DNS server(s).

This can be due to the Supervisor cluster's Management and/or Workload Network having the wrong DNS server configuration.

Changes to the Supervisor cluster's DNS server configuration may not have propagated to the systemctl service serving DNS in one or more Supervisor VMs.

Resolution

See if the DNS servers for the Supervisor cluster's Management and Workload Network configuration need correction.
- This is found in the vSphere web UI under the Supervisor cluster -> Configure -> Network
Check that the configured DNS servers are functioning properly.
Once it is confirmed that the DNS server configuration is correct and there are no issues with the DNS servers, restart the systemd-resolved systemctl service to pick up the correct DNS servers:
```
systemctl restart systemd-resolved
```

Ensure that the DNS server IPs reflect correctly in the systemd-resolved journal logs:

journalctl -xeu systemd-resolved

Using degraded feature set UDP instead of TCP for DNS server <correct DNS server IP>.
Using degraded feature set TCP instead of UDP for DNS server <correct DNS server IP>.
Using degraded feature set UDP instead of UDP+EDNS0 for DNS server <correct DNS server IP>.

Confirm that all Supervisor VMs are able to reach the public Broadcom repository:
- See the below KB for detailed steps:
  Networking Requirements for downloading VKS Supervisor Services

Additional Information

This issue can similarly occur for environments using private registries if the private registry is unable to be reached by one or more Supervisor VMs due to the above DNS issue.