TMC deployment fails with ERR Failed to install TMC-SM error="install execution error: step relocate images running error: error pushing

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

TMC SM deployment failing with error like below. 

ERR Failed to install TMC-SM error="install execution error: step relocate images running error: error pushing images: parallel agent image push failed: work item with input
...
UNKNOWN: unknown error; map[DriverName:filesystem Enclosed:map[Err:28 Op:mkdir Path:/storage/docker/registry/v2

Environment

vSphere Supervisor
TMC SM

Cause

Space full on Harbor registry
Confirmed by checking the pod from the supervisor

kubectl -n svc-harbor-domain-<harbor domain> exec -it harbor-registry-<pod id> -- df -h | sort -hr | head -10
Defaulted container "registry" out of: registry, registryctl
tmpfs           768M 4.0K  768M   1% /etc/registry/passwd
tmpfs           768M  12K  768M   1% /etc/harbor/ssl/registry
tmpfs            64M    0   64M   0% /dev
tmpfs           363M    0  363M   0% /sys/fs/cgroup
shm              64M    0   64M   0% /dev/shm
overlay         253M  83M  169M  33% /
Filesystem      Size Used Avail Use% Mounted on
/dev/sdb         10G  10G     0 100% /storage
/dev/sda        253M  83M  169M  33% /dev/termination-log

 

Resolution

Increase space for registry pod. If Harbor deployed as supervisor service. Update the yaml file for the supervisor service from the vSphere UI to desired state and save. 

Workload management > Supervisor > Services > Harbor service > Edit > increase registry PVC in values file.