Harbor Pod failed to mount a PVC

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After restarting a harbor pod, it was on pending status, because it failed to mount a pvc

Environment

vSphere with Tanzu 8.x

Cause

After restarting a harbor pod, the old pod was stuck on the worker node, and didn't release the volume

Resolution

1. Check the volumeattachement status and which node it belong to 

 kubectl get volumeattachments.storage.k8s.io

2. SSH to the target node

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vsphere-supervisor-services-and-standalone-components/latest/managing-vsphere-kuberenetes-service-clusters-and-workloads/configuring-identity-and-access-for-tkg-service-clusters/connect-to-the-tkg-service-cluster-control-plane-as-a-kubernetes-administrator/ssh-to-tkg-cluster-nodes-as-the-system-user-with-a-password.html

3. Check the mount status via lsblk and df command.

df -h

lsblk

4. Check the containers status

crictl pods 

5. If the container is gone, we can use the unmount command to release the volume. detached it from VM

6. If the container is still running, we can use crtctl command to stop the pod.