Harbor Replication fails in the Workload Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Harbor replication in the Workload Cluster fails.
• The following error message is displayed when clicking the information icon for a "Failed" status in the Replications menu:

  failed to create adapter for destination registry https://<XXX>: Get "https://<XXX>/api/version": dial tcp: lookup XXX on XXX:53 server misbehaving

• Although ping (ICMP) connectivity from the Workload Cluster node to the DNS server is successful, name resolution communication using traceroute (UDP), dig, or nslookup commands fails.

  vmware-system-user@XXX-XXX-XXX [ ~ ]$ ping XXX.XXX.XXX.XXX
  PING XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) 56(84) bytes of data.
  64 bytes from XXX.XXX.XXX.XXX: icmp_seq=1 ttl=127 time=0.556 ms
  64 bytes from XXX.XXX.XXX.XXX: icmp_seq=2 ttl=127 time=0.818 ms
  64 bytes from XXX.XXX.XXX.XXX: icmp_seq=3 ttl=127 time=0.604 ms
  
  vmware-system-user@XXX-XXX-XXX [ ~ ]$ traceroute XXX.XXX.XXX.XXX
  traceroute to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX), 30 hops max, 60 byte packets
   1  XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX)  0.261 ms !X  0.331 ms !X  0.341 ms !X
  
  vmware-system-user@XXX-XXX-XXX [ ~ ]$ dig XXX @XXX.XXX.XXX.XXX
  ;; communications error to XXX.XXX.XXX.XXX#53: host unreachable
  ;; communications error to XXX.XXX.XXX.XXX#53: host unreachable
  ;; communications error to XXX.XXX.XXX.XXX#53: host unreachable
  
  ; <<>> DiG 9.20.0 <<>> XXX @XXX.XXX.XXX.XXX
  ;; global options: +cmd
  ;; no servers could be reached
  
  vmware-system-user@XXX-XXX-XXX [ ~ ]$ nslookup XXX XXX.XXX.XXX.XXX
  ;; communications error to XXX.XXX.XXX.XXX#53: host unreachable
  ;; communications error to XXX.XXX.XXX.XXX#53: host unreachable
  ;; communications error to XXX.XXX.XXX.XXX#53: host unreachable
  ;; no servers could be reached

Environment

VMware vSphere Kubernetes Service

Cause

The !X displayed in the traceroute command output indicates "Communication Administratively Prohibited," meaning that traffic is being rejected by network devices such as firewalls or routers along the path.

Resolution

Check the configuration of network devices along the path from the network where the Workload Cluster nodes (virtual machines) are deployed (Workload Network) to the DNS server.

Additional Information

Japanese KB: Workload Cluster の Harbor の Replication が失敗する