Error: "failed to resolve reference "projects.registry.vmware.com/vmware-cloud-director/tanzu-packages-airgapped-v2.4.0:4.1.1"" causing Kubernetes Cluster creation failures

Products

• VMware Cloud Director

Issue/Introduction

• Attempting to create Kubernetes clusters fails.
• Cannot create Kubernetes clusters using the Tanzu Kubernetes Grid (TKG) 2.4.0 Ubuntu 2004 K8s OVA templates through Container Service Extension (CSE) in Cloud Director (VCD).
• The EPHEMERAL-TEMP-VM is deployed and powered on but the /var/log/cloud-final.err log shows a failure to download required packages from the configured container registry:

failed to resolve reference "projects.registry.vmware.com/vmware-cloud-director/tanzu-packages-airgapped-v2.4.0:4.1.1": projects.registry.vmware.com/vmware-cloud-director/tanzu-packages-airgapped-v2.4.0:4.1.1: not found

• Creating Kubernetes clusters succeeds when using older TKG versions such as 2.2.0.

Environment

VMware Cloud Director
VMware Cloud Director Container Service Extension 4.1.1

Cause

This is expected behaviour as CSE 4.1.1 does not support TKG 2.4.0.

The container registries, projects.registry.vmware.com and projects.packages.broadcom.com, do not contain packages for the EPHEMERAL-TEMP-VM to download for the CSE 4.1.1 release.
Packages only exist for later CSE 4.2.x versions.

Resolution

To resolve this issue, use the appropriate TKG versions supported by CSE 4.1.1 as per the release notes, VMware Cloud Director Container Service Extension 4.1.1a Release Notes - Compatibility Updates.

Alternatively to use TKG 2.4.0 templates upgrade CSE to a valid 4.2.x release which supports the desired TKG versions detailed in the VMware Product Interoperability Matrix.
Follow the upgrade steps in the release notes of the desired version to upgrade both the Kubernetes Container Clusters UI plug-in and the CSE Server.

Additional Information

To verify the CSE Server version, SSH to the CSE Server as root and run systemctl status cse.service.
The current CSE Server version will be displayed, for example:

cse.service - Cloud Director Container Service Extension 4.1.1