Error: "Temporary failure resolving [Hostname]" during nvidia-driver-daemonset during PAIS 2.1 deployment

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Deploying models for PAIS 2.1 does not complete. When you look at the nvidia-driver-daemonset pod logs the follow errors are found:
  
  W: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/InRelease  Temporary failure resolving 'developer.download.nvidia.com'

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/noble/InRelease  Temporary failure resolving 'archive.ubuntu.com'


• Checking underlying DNS infrastructure of the environment and VKS with CoreDNS show no routing or DNS issues.
  
  
• Further investigation of the host NIC for errors show some queues with TE pkt errors:
  
  /usr/lib/vmware/vm-support/bin/nicinfo.sh | grep "TE pkt errors tx"

      [txq0] TE pkt errors tx: 4554
      [txq1] TE pkt errors tx: 0
      [txq2] TE pkt errors tx: 0
      [txq3] TE pkt errors tx: 0
      [txq4] TE pkt errors tx: 330
      [txq5] TE pkt errors tx: 0
      [txq6] TE pkt errors tx: 0
      [txq7] TE pkt errors tx: 0
      [txq8] TE pkt errors tx: 0
      [txq9] TE pkt errors tx: 0

Environment

• PAIS 2.1
• VCF 9.1
• NSX 

Cause

The BCM5760 NIC's for the host will run into packet drops due to issues with tagging and untagging when NSX is using double encapsulation. This behavior causes false positive errors such as DNS lookup issues.

Resolution

The resolution is discussed in this article:

VKS ESXi host in the guest cluster with BCM5760x NIC drop double encap geneve traffic