Error: Supervisor performance is erratic, vmop, imageregistry, and etcd are affected.
search cancel

Error: Supervisor performance is erratic, vmop, imageregistry, and etcd are affected.

book

Article ID: 445059

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

Supervisor performance is erratic and may become unresponsive when the default content library is assigned  across many individual namespaces.

Symptoms include:

  • VM Operator (VMOP) and Image Registry pods are in a CrashLoopBackOff state with OOMKilled errors.
  • etcd  database shows excessive growth (bloat) and slow response times.
  • The Image Registry operator crashes regularly.
  •  High memory consumption in the **vmware-system-vmop-controller-manager** pod due to managing an excessive number of `VirtualMachineImage` objects.

Environment

vCenter 8.x

vSphere with Kubernetes

Cause

This issue occurs when a customer has many namespaces and a Kubernetes Service Content Library is manually associated with individual vSphere Namespaces, even though it is already defined as the Default Content Library at the Supervisor level.

The default subscription  Content Library has over 120 objects. By assigning it to every namespace a key is generated for each and stored in etcd.

Similarly this associates a number of VirtualMachineImage objects to the namespace and  a key is generated for each and stored in etcd.

When associated at the namespace level, the Supervisor propagates a duplicate set of VirtualMachineImage objects and imageregistry.vmware.com objects for every namespace.

As the number of namespaces grows, this redundancy leads to a massive accumulation of keys in etcd, with high memory consumption  overwhelming the controller memory limits during reconciliation.

 

Resolution

1. Verify the Redundancy

Check the name of the content library configured for the Supervisor

In the vSphere Client, Navigate to  Workload Management  >  Supervisors  >  Configure  >  Tanzu Kubernetes Grid Service  >  Content Library .

Note the library defined here.

2. Disassociate the Library

In the vSphere Client, Remove the Content Library from each Namespace where it has been added.

Note: The VM Service card should only contain content libraries for custom/standalone VM images. If the namespace only uses standard TKG images, it should have 0 associated libraries.

3: Stabilize System Pods (If Necessary)

If the pods do not recover automatically after the objects are removed, manually restart using  kubectl to delete the affected pods.

Note 

    • Resolution Strategy: Disassociate redundant Content Libraries from individual vSphere Namespaces.
    • Impact: Non-disruptive,
      • As the Supervisor Content Library  is unchanged,  the ClusterContentLibrary already exists and so a rollout is not triggered.
      • As the images are already present as ClusterVirtualMachineImage objects (backed by the Supervisor-level Content Library), removing the namespace-level association does not change the VM spec or image resolution path for existing guest clusters.
    • Verification: Confirm kubectl get cvmi shows the required images remain available and READY: True after the namespace libraries are removed.

If the etcd's keyspace data limit has been reached or exceeded as per vSphere Kubernetes Cluster Unhealthy, Kubectl Commands Failing due to ETCD Database Full or Exceeded

Please open a ticket to VMware by Broadcom Technical Support referencing this KB for assistance in cleaning up ETCD database and restoring it to operational state.

Additional Information

For further guidance on managing pod memory or cleaning up stale items, see the following resources:

Powered by Wolken Software