nslookup kube-dns.kube-system.svc.cluster.local failed with connection timed out

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• New ESXi hosts added to Supervisor cluster.
• Name resolution does not work with Supervisor cluster intermittently:
  
  # nslookup kube-dns.kube-system.svc.cluster.local
connection timed out; no servers could be reached
  
  
• The issue happens when any of coredns pods running on the new added ESXi hosts.
• The uplink of new ESXi host Broadcom BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller with firmware and driver 220.0.9.0 / 22.31.13.70

Environment

vSphere with Tanzu

Resolution

• To fix the issue, contact network adapter vendor to upgrade firmware and driver to latest version, such as FW 23.21.13.39 and Driver 232.0.154.0
• To workaround the issue, software simulation of Geneve Offload from hardware to software with the command:
  
  esxcli network nic software set --vmnic=vmnicX --geneveoffload=1

More information, see Wrong GENEVE inner checksum from NIC firmware on ESXi host with BCM5741x / Broadcom 5741x NIC