OVF Templates Deleted from Content Library Linked to Multiple vSphere Namespaces

Products

VMware vCenter Server VMware vSphere 7.0 with Tanzu VMware vSphere with Tanzu vSphere with Tanzu Tanzu Kubernetes Runtime

Issue/Introduction

In Workload Management (WCP) and vSphere Kubernetes Service (VKS), when the same local content library is associated with multiple namespaces, deleting any of those namespaces or unlinking the local content library from one of the namespaces will lead to the deletion of all templates from that local content library.

All OVF templates in the local content library are deleted, resulting in missing images in the Supervisor cluster.
This can also occur if one of the namespaces using the local content library is deleted.
Changes or removal of the local content library can result in the above issues regardless of whether the local content library was attached on the Tanzu Kubernetes Grid Service or VM service.
Any workflows dependent on the deleted templates will be disrupted.
- This includes creation of new nodes, preventing cluster creation, scaling up, upgrade or other changes requiring rolling redeployment of vSphere Kubernetes clusters.

Environment

VMware vCenter Server 8.0.2

vSphere with Tanzu 8.0

vSphere Kubernetes Supervisor 8.0

Cause

For a local content library associated with multiple namespaces in a Supervisor cluster, if one namespace is deleted or unlinks the local content library, it will also affect other namespaces in the Supervisor cluster.

This only impacts a local content library that is used for Tanzu Kubernetes Grid Service or VM Service. Other local content libraries are not affected by this issue.
If the impacted local content library is associated at the Tanzu Kubernetes Grid Service, it will impact the lifecycle of all existing Tanzu Kubernetes Clusters (TKCs)

Deletion or removal of the local content library from one of the multiple namespaces using the same local content library will result in deletion of OVF templates that were uploaded to the same local content library.

When OVF templates are missing, the Supervisor cluster will not be able to populate available cluster virtual machine images or Kubernetes Releases.

Without available cluster virtual machine images or Kubernetes Releases, new clusters and nodes cannot be provisioned.

This impacts node creation including new clusters, scaling node operations and rolling redeployments.

This is a known bug introduced by the Image Registry Service feature starting in vCenter 8.0u2.

vSphere Supervisor 8.0 Release Notes under "Templates might get deleted from the Content Library in vCenter when the library is linked to multiple vSphere namespaces"

Resolution

The issue was fixed in the vCenter 8.0.2 MP1 release.

Because the fix is part of the Image Registry Operator component which runs on the Supervisor cluster, the Supervisor cluster must be upgraded to implement the fix.

Workaround

High-Level Overview:

Publish the local content library
Create a subscribed content library
Subscribe the content library to the published local content library
Link the Supervisor cluster namespaces to the new subscribed content library

This will prevent changes made to the content library in one namespace, or the deletion of a namespace from deleting OVF templates in the content library.

Follow documentation to publish the local content library and create at least one content library that is subscribed to the published local content library:
- Enable Publishing of a Local Content Library
Ensure that the published local content library is populated with the desired OVF Templates.
Associate the new content library that is subscribed to the published local content library with the Supervisor cluster's namespaces:
- WARNING: This will result in an automatic rolling redeployment of all existing vSphere Kubernetes clusters that use this content library.
- For the Tanzu Kubernetes Grid service, see "Associate the TKR Content Library with the TKG Service" from Configure a vSphere Namespace for TKG Service Clusters
  - Note: Content library changes made to the Tanzu Kubernetes Grid Service will also reflect on all other namespaces in the Supervisor cluster and result in the rolling redeployment of all clusters across all namespaces.
- For the VM service, see Associate a VM Content Library with a Namespace in vSphere with Tanzu
To confirm that the Supervisor cluster recognizes the new subscribed content library, the following steps can be performed:
- Connect to the Supervisor cluster context
- Confirm that the contentsource matches the content library ID of the new subscribed content library:
  - ```
  kubectl get contentsources -A
```
- Check that the cluster virtual machine images (cvmi) are populated:
  - ```
  kubectl get cvmi
```
- Confirm Tanzu Kubernetes Releases (TKR) or Kubernetes Releases (KR) are present:
  - ```
  kubectl get tkr,kr
```

Additional Information

The image registry operator pod on Supervisor control plane would contain the following log which indicates that the library item was deleted from the Content Library in vCenter, where the values enclosed in <> vary by environment:

YYYY-MM-DDTHH:MM:SS.sssssssssZ stderr F IMMDD HH:MM:SS.ssssss 1 contentlibraryitem_controller.go:351] controllers/ContentLibraryItem "msg"="Deleted library item in VC" "ID"="<content library id>" "clItemName"="<content library item id>"