PKS Clusters Fail to Pull Harbor Images

Products

• VMware Cloud PKS

Issue/Introduction

Symptoms:

• PKS clusters fail to pull images from the Harbor registry. 
• You see messages similar to the following kubelet.stderr.log from the worker node.

E0112 19:28:39.804868   13305 kuberuntime_image.go:51] Pull image "Harbor_FQDN.domain.local/library/my_image:lastest" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://Harbor_FQDN.domain.local/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
 

• You see messages similar to the following in the docker.stderr.log file on the worker node:

time="2020-01-12T19:28:39.803914271Z" level=error msg="Handler for POST /v1.38/images/create returned error: Get https://Harbor_FQDN.domain.local/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
 

• All services show running for harbor. Errors are not seen in the Harbor logs. Docker instances in subnets outside of the PKS node IP pool might be able to pull images.

Environment

VMware PKS 1.x

Cause

Generally this is caused by a network issue. One can be that the internal container network that Harbor uses overlaps with the PKS cluster node IP pool and so responses stay internal to Harbor instead of leaving back to the PKS cluster worker node.

Resolution

Validate that the internal container IPs are overlapping the PKS cluster worker IPs. Use bosh ssh to ssh to the harbor-app. Then run the following.

sudo su -
cd /var/vcap/packages/docker/bin/
./docker -H unix:///var/vcap/sys/run/docker/dockerd.sock network inspect $(./docker -H unix:///var/vcap/sys/run/docker/dockerd.sock network ls -q)|grep -B1 "Gateway" 

If the PKS cluster and Harbor internal networks overlap then set the "Address pool1 base" to a subnet that does not overlap with the node IP block. This is done in the OpsMan UI in the Harbor Tile Networking for customized container network settings. For details  See the section "Configure Networking " in the harbor documentation. 

The PKS node IP block range can be checked in the NSX-T UI in the IPAM settings under the advanced network menu settings.