vmware-system-imageregistry-controller-manager pods are in in CrashLoopBackOff state with the "error: invalid memory address or nil pointer dereference"

Products

• vSphere with Tanzu

Issue/Introduction

• Guest cluster is struck in updating state
• Supervisor cluster's vmware-system-imageregistry-controller-manager pods are in the CrashLoopBackOff state with "error: invalid memory address or nil pointer dereference"

$ kubectl logs -n vmware-system-imageregistry vmware-system-imageregistry-controller-manager-abcdefabcb-abcd
:
I0905 03:42:09.019583       1 contentlibrary_controller.go:274] controllers/ContentLibrary "msg"="Processing library item" "clName"="cl-ae79201c6d511d310" "itemID"="1c3b5646-e261-4d5f-b4d0-560aecad9bd9"
I0905 03:42:09.019606       1 controller.go:118]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "ContentLibraryItem"={"name":"clitem-971cc0ec130eabcde","namespace":"tkg-cluster"} "controller"="contentlibraryitem" "controllerGroup"="imageregistry.vmware.com" "controllerKind"="ContentLibraryItem" "name"="clitem-971cc0ec130eabcde" "namespace"="tkg-cluster" "reconcileID"="3c474adb-d397-4011-91c9-08c37e55c72b"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]

$ kubectl logs -n vmware-system-imageregistry vmware-system-imageregistry-controller-manager-abcdefabcb-abcd
:
Events:
  Type     Reason   Age                      From     Message
  ----     ------   ----                     ----     -------
  Warning  BackOff  8m28s (x11499 over 42h)  kubelet  Back-off restarting failed container manager in pod vmware-system-imageregistry-controller-manager-abcdefabcb-abcd_vmware-system-imageregistry(d160c57b-9cf1-4cda-b7ec-e5be700d41e8)
  Normal   Pulled   3m22s (x468 over 42h)    kubelet  Container image "localhost:5000/vmware/imageregistry-operator@sha256:bc9b2579e3ecff338303d8b11b9af253ba3fe71da6462a6d1bf2e9a51180b3a8" already present on machine

Environment

vSphere with Tanzu 

vCenter Version: 8.0U2  

VMware ESXi Version: 8.0.2 

NSX-ALB Version: 22.1.5

Supervisor Version: v1.26.x

Guest Cluster Version: v1.26.x

Cause

The kubectl logs shows that this was getting redirected to stale contentlibraryitem of clitem-971cc0ec130eabcde in this example. 

# kubectl logs -n vmware-system-imageregistry vmware-system-imageregistry-controller-manager-abcdefabcb-abcd

:
I0905 03:42:09.019583       1 contentlibrary_controller.go:274] controllers/ContentLibrary "msg"="Processing library item" "clName"="cl-ae79201c6d511d310" "itemID"="1c3b5646-e261-4d5f-b4d0-560aecad9bd9"
I0905 03:42:09.019606       1 controller.go:118]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "ContentLibraryItem"={"name":"clitem-971cc0ec130eabcde","namespace":"tkg-clusters"} "controller"="contentlibraryitem" "controllerGroup"="imageregistry.vmware.com" "controllerKind"="ContentLibraryItem" "name"="clitem-971cc0ec130eabcde" "namespace"="tkg-clusters" "reconcileID"="3c474adb-d397-4011-91c9-08c37e55c72b"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x174d98c]

Resolution

• Check for stale clitems by running the below command and notice the blank items without any contenlibraryreference

# kubectl -n tkg-clusters get clitem
NAME                      VSPHERENAME CONTENTLIBRARYREF  TYPE  READY  CACHED  SIZEINBYTES  SECURITYCOMPLIANT  AGE
clitem-06412b08e23302008                                                                                                                                                                                                                                                    3d9h
clitem-1138df6b037bf1048                                                                                                                                                                                                                                                     3d9h
:
:

 

• Run the below command to patch the clitem for all the empty items

# kubectl -n <namespace> patch contentlibraryitem <clitem-name> -p '{"metadata":{"finalizers":null}}' --type merge

Example:
# kubectl -n tkg-clusters patch contentlibraryitem clitem-06412b08e23302008 -p '{"metadata":{"finalizers":null}}' --type merge
# kubectl -n tkg-clusters patch contentlibraryitem clitem-1138df6b037bf1048 -p '{"metadata":{"finalizers":null}}' --type merge

 

• Restart the content-library services on the vCenter