"Unexpected EOF" error when pulling images from VIC Appliance registry

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When you try to pull any image from the VIC Appliance Harbor registry using a VCH (Virtual Container Host), you receive the following error:

root@PhotonJumpBox [ ~/vic ]# docker -H VCH_IP:2376 --tls pull VIC_APPLIANCE_FQDN/default-project/alpine
Using default tag: latest
latest: Pulling from default-project/alpine
e6b0cf9c0882: Pulling fs layer
a3ed95caeb02: Downloading [==================================================>]      32B/32B
default-project/alpine/cc327eed37d32c79a81925daeedce436c929741892ca53c4b6abbe19e03acf97 returned download failed: unexpected EOF

• When you try to pull any image from dockerhub using a VCH, it works fine :

root@PhotonJumpBox [ ~/vic ]# docker -H 192.168.100.50:2376 --tls pull alpine
Using default tag: latest
latest: Pulling from library/alpine
c9b1b535fdd9: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45
Status: Downloaded newer image for library/alpine:latest

•  While using a Docker client, you can pull the images from VIC Appliance registry.

root@PhotonJumpBox [ ~/vic ]# docker pull vicappliance00.gsslabs.org/default-project/alpine
Using default tag: latest
latest: Pulling from default-project/alpine
e6b0cf9c0882: Pull complete
Digest: sha256:3983cc12fb9dc20a009340149e382a18de6a8261b0ac0e8f5fcdf11f8dd5937e
Status: Downloaded newer image for vicappliance00.gsslabs.org/default-project/alpine:latest

• In /var/log/vic/docker-personality.log on the VCH, you can see similar entries:

Feb 17 2020 15:46:04.908Z ERROR Fetch (https://VIC_APPLIANCE_FQDN/project/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4) to file failed to stream to file: unexpected EOF
Feb 17 2020 15:46:04.908Z DEBUG op=2308.200: [ END ] [VIC_APPLIANCE_FQDN/pkg/fetcher.(*URLFetcher).fetchToFile:324] [104.805863ms] 
https://VIC_APPLIANCE_FQDN/project/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Feb 17 2020 15:46:04.908Z DEBUG Error: download failed: unexpected EOF
Feb 17 2020 15:46:04.908Z DEBUG op=2308.199: [ END ] [VIC_APPLIANCE_FQDN/pkg/fetcher.(*URLFetcher).Fetch:138] [104.851519ms] 
https://VIC_APPLIANCE_FQDN/project/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

You can access the VCH logs either using the VCH Admin page : https://VCH_IP:2378
Or you can enable SSH on the VCH, using the vic-machine-OS command line, using the following steps on the documentation below:

• https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/vch_shell_access.html

Environment

VMware vSphere Integrated Containers 1.x

Cause

This issue occurs when the VIC Appliance had the Garbage Collector enabled, and it has been rebooted.
At VIC Appliance boot time (or when restarting the Harbor service), the Garbage Collector deletes that layer from the File System, which is used for backward compatibility of Docker images, but not from its cache.
All the VCHs which will try to load new images from this registry will fail.

Resolution

To resolve this issue, we have to manually re-create that image layer, following steps below:

1. Download the attached file named "data.zip", from this KB article.
2. Unzip the file "data.zip"
3. Copy this file into the VIC Appliance, using your preferred tool (scp, WinSCP, etc...)
4. SSH onto the VIC Appliance using your preferred tool (ssh, PuTTY, etc...)
5. Check that the SHA256 checksum of the file matches the following :

root@vicappliance00 [ ~ ]# sha256sum data
a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4    data

5. Create the folder structure using:

mkdir -p /storage/data/harbor/registry/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 

6. Move the file named data into the newly created folder

mv data /storage/data/harbor/registry/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4

Workaround:

1. Back up images from vSphere Integrated Container Appliance localy on a docker client

docker pull VIC_APPLIANCE_FQDN/PROJECT_NAME/IMAGE:TAG

2. Deploy new vSphere Integrated Container Appliance, and recreate all Projects, and different configurations
3. Push backed up images from Docker client to new vSphere Integrated Container Appliance

docker tag OLD_VIC_APPLIANCE_FQDN/PROJECT_NAME/IMAGE:TAG NEW_VIC_APPLIANCE_FQDN/PROJECT_NAME/IMAGE:TAG
docker push NEW_VIC_APPLIANCE_FQDN/PROJECT_NAME/IMAGE:TAG