vSphere Supervisor Workload Cluster Nodes Recreated Continuously due to duplicate Content Library TKRs

Products

VMware vSphere Kubernetes Service VMware vSphere 7.0 with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

Nodes in the affected vSphere Supervisor Workload cluster are recreated in a loop every 60 to 120 minutes.

While connected to the Supervisor cluster context, the following symptoms are observed:

The affected workload cluster's machines associated with the recreating node are reaching Running state but are being recreated over and over:
- ```
kubectl get machines -n <affected workload cluster namespace>
```
The recreation loop occurs independently for each affected node, similar to a rolling redeployment.
- This means that the nodes are not all being recreated at the same time/simultaneously.
The affected recreating machines remain in Running state until the recreation loop occurs.
All of the affected workload cluster's worker nodes and control plane nodes are stuck in this recreation loop.

There are duplicate virtual machine images on the same OS in the environment:

The below example has two photon OS images for an example TKR, one of which has additional alphanumerics appended:
- The CONTENTSOURCENAME corresponds to the ID of a content library. In the below example, this photon osimage is present in two separate content libraries, creating a duplicate virtualmachineimage ending in tkd7np.

kubectl get virtualmachineimages


NAME              CONTENTSOURCENAME                      VERSION           OSTYPE                FORMAT   AGE
ob-<id>-tkgs-ova-photon-X-<TKR version>---vmware.X-fips.X-tkd7np   <content-library-id-a>   <TKR version>+vmware.X-fips.X-tkg.X   vmwarePhoton64Guest   ovf      XmXXs

ob-<id>-tkgs-ova-photon-X-<TKR version>---vmware.X-fips.X-tkg.X <content-library-id-b>   <TKR version>+vmware.X-fips.X-tkg.X   vmwarePhoton64Guest   ovf      XmXXs

There may be multiple content libraries containing the same TKR osimages attached to any namespace in the Supervisor cluster environment:

```
kubectl get contentsources -A
```
The above IDs correspond to the ID of each content library which can be found in the vSphere web client.

From the vSphere web Client, there is a content library assigned to the Tanzu Kubernetes Grid Service card and on the VM Service card of any namespace.

Both assigned content libraries contain identical TKR osimages.
It is not recommended to have a content library with the same TKR images attached to the VM Service card.
This can also occur if the same content library is attached to both the Tanzu Kubernetes Grid Service card and the VM service card on any namespace.

This duplicate image issue can also occur if there is only one content library attached to the Tanzu Kubernetes Grid Service card but this content library contains duplicate TKR osimages.

The content library attached to the Tanzu Kubernetes Grid Service card is shared across all namespaces in the Supervisor cluster environment.

Environment

vSphere 7.0 with Tanzu

This issue can occur regardless of whether or not this cluster is managed by Tanzu Mission Control (TMC)

Cause

The environment attempts to reconcile and pull images from the content libraries associated with the namespaces in the Supervisor cluster environment.

However, if a content library containing the same TKR osimages is attached to both the Tanzu Kubernetes Grid Service and VM Service card in the namespace, this will result in duplicate virtualmachineimages in the environment. This can also occur if the same content library is attached and assigned in both of the above cards.

In vSphere 7.X, the affected workload cluster nodes will continue to recreate because of the conflicting duplicate images from the above noted content libraries. The system will continue to try to deploy nodes by pulling from the associated content libraries but remain stuck in a loop due to the duplicate images.

This duplicate image issue can also occur if there is only one content library attached to the Tanzu Kubernetes Grid Service card but this content library contains duplicate TKR osimages.

The content library attached to the Tanzu Kubernetes Grid Service card is shared between all namespaces in the Supervisor cluster environment.

Note: Due to a race condition in vSphere 7.x, duplicate images can also occur despite there being no duplicate images in the content library and despite the content library being configured properly.

This issue is resolved in vSphere 8.0u2 and higher.

Resolution

The duplicate virtualmachineimages will need to be cleaned up from the Supervisor cluster environment. This involves checking for multiple content libraries with the same TKR osimages assigned to the same namespace, duplicate TKR osimages within the same content library and removing the duplicate reference or images from the Supervisor cluster environment.

Caution: When modifying/deleting a Content Library from vCenter, please make sure it's not being actively used by any other Supervisor cluster, TKC cluster, etc.

Modifying or deleting any content library in use can cause a rolling redeployment of the nodes using the content library's images. This is because the system must recreate the nodes to use an image that is from a different content library.

Steps for Duplicate Content Library on VM Service card

While connected to the Supervisor cluster context, ensure that the affected workload cluster(s) are not set to "paused:true":
- ```
kubectl get cluster -o yaml -n <workload cluster namespace> <workload cluster name> | grep -i "paused"
```
- If the workload cluster shows "paused:true", please reach out to VMware by Broadcom Technical Support referencing this KB article.
Please see the following KB article: TKGS Cluster is rolling continuously. VMOP pod logs show "VirtualMachine "msg"="patch failed" " (87973)

Steps for Duplicate Content Libraries in the Environment

On vSphere Client navigate to Content Libraries:

Examine each of the Content Libraries listed containing TKRs.
Click on the Content Library name -> Templates -> OVF & OVA Templates

If the same TKRs are included in different Content Libraries there are two options:

1. Remove the duplicated TKRs (OVF & OVA templates) from all Content Libraries except one.
2. Remove the duplicated Content Libraries and leave just one Content Library with TKRs in it.

Note: The only Content Library containing TKRs should be configured in the Supervisor Cluster's Tanzu Kubernetes Grid Service:

(Workload Management -> Supervisors -> Select the Supervisor Cluster -> Configure -> General -> Tanzu Kubernetes Grid Service -> Content Library)

In the example above, as duplicated TKRs exist in both Content Libraries, we will delete "TKG-CL-2" and leave "TKG-Content-Library" configured in TKG Service for the existing Supervisor Cluster.

To remove the Content Library:

* Select the Content Library:

* Click on Actions -> Delete

* Select "Yes" after the Warning message shows up.

* Go back to Content Libraries and verify that it has been deleted successfully.

Now that only one Content Library with TKRs is left in vCenter, verify that it's assigned in the Supervisor Cluster's TKG Service:

Workload Management -> Supervisors -> Select the Supervisor Cluster -> Configure -> General -> Tanzu Kubernetes Grid Service -> Content Library

Confirm that there are no longer any duplicate images within the Supervisor cluster environment

Connect to the Supervisor cluster context
Check for any duplicate virtualmachineimage entries:
- ```
kubectl get virtualmachineimage
```

Restart the VMOP pods:

kubectl rollout restart deploy -n vmware-system-vmop vmware-system-vmop-controller-manager

Perform a deletion on any duplicate virtualmachineimage entry:
- WARNING: Any workload clusters using the duplicate image will result in a rolling redeployment of that cluster.
- ```
kubectl delete virtualmachineimage <duplicate image with extra alphanumerics>
```
- If there is no content library with duplicate TKR osimages configured for any of the namespaces under the Supervisor cluster, the duplicate virtualmachineimage should not re-appear.
Ensure that the affected workload cluster(s) are not set to "paused:true":
- ```
kubectl get cluster -o yaml -n <workload cluster namespace> <workload cluster name> | grep -i "paused"
```
- If the workload cluster shows "paused:true", please reach out to VMware by Broadcom Technical Support referencing this KB article.

Additional Information

This issue is resolved in vSphere 8.0u2 and higher.

Impact/Risks:

Caution: When modifying/deleting a Content Library from vCenter, please make sure it's not being actively used by any other Supervisor cluster, TKC cluster, etc.