docker-registry pod in ContainerCreating state and "hostPath type check failed: /storage/container-registry is not a directory"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Scenario:

• One or more of the docker-registry pods on the Supervisor nodes are stuck in ContainerCreating state

Example:

kube-system   docker-registry-###############################                  1/1     Running             0               529d    ##.##.##.##   ###############################   <none>           <none>
kube-system   docker-registry-###############################                  0/1     ContainerCreating   0               6h51m   ##.##.##.##     ###############################   <none>           <none>
kube-system   docker-registry-###############################                  1/1     Running             0               529d    ##.##.##.##   ###############################   <none>           <none>

 

• Describing the failing pod shows error:

Warning  FailedMount  7m50s (x168 over 5h33m)  kubelet  MountVolume.SetUp failed for volume "storage-container-registry" : hostPath type check failed: /storage/container-registry is not a directory

 

• When you ssh to the Supervisor control plane node and check for that directory it does not exist:

$ ls -l /storage/container-registry

ls: cannot access '/storage/container-registry': No such file or directory

Environment

VMware vSphere Kubernetes Service

Cause

If there is a system-related issue or the registry was created manually or recreated, this behavior is by design:

The type: Directory option enforces that the path must already be a directory on the host; it will not create the directory for you.

If you want Kubernetes to create the directory automatically if it does not exist, you must use type: DirectoryOrCreate instead.

Resolution

1. ssh to the affected Supervisor control plane node
   see "How to SSH into Supervisor Control Plane VMs" in Troubleshooting vSphere with Tanzu (TKGS) Supervisor Control Plane VMs
   
   
2. Create the missing directory
   # sudo mkdir -p /storage/container-registry


3. Make sure the permissions are set correctly
   
   # chmod 0755 /storage
# chmod 0755 /storage/container-registry


4. Run the following command to create all the subdirectories under  "/storage/container-registry" directory and sync the images from a health Supervisor control plane node.
   
   # /usr/lib/vmware-wcp/upgrade/upgrade-ctl.py sync-images --source <CPVM-Management-IP-Address>
   
   note: we need to sync the images using the CPVM IP address  from Management network not the one on the Workload network
   
   
5. Restart the docker-registry pod
   
   # kubectl delete pod -n kube-system  docker-registry-###############################