New content libraries remain in Not Ready state in VCFA due to Image Registry operator reconciliation failure

Products

• VCF Automation

Issue/Introduction

• In VMware Cloud Foundation (VCF), you may observe that newly created content libraries are stuck in a Not Ready state. Despite the vmware-system-imageregistry pods showing a Running status in the cluster, they stop reconciling new content.
• Manual attempts to add or remove libraries do not trigger any system response, and logs may show a lack of telemetry or update activity since a specific timestamp.

Environment

• Product: VMware Cloud Foundation (VCF) / VMware vSphere with Tanzu

• Component: Image Registry Operator (vmware-system-imageregistry)

Cause

• The Image Registry controller manager entered an unresponsive state (software hang). Because the Kubernetes liveness probes did not detect the deadlock, the container remained "Running" but the synchronization loop between the cluster and the vCenter Fabric Assistant (VCFA) was halted.

Resolution

• To restore the reconciliation loop and synchronize the Virtual Machine Images (VMIs), you must restart the deployment to clear the process deadlock:

1. 1. Log in to the Supervisor Cluster via CLI.

   2. Execute a rolling restart of the Image Registry deployment:
      
      kubectl rollout restart deployment -n vmware-system-imageregistry <deployment-name>

   3. Monitor the new pods to ensure they reach a Running state and verify that content libraries transition to Ready.