Kubernetes (K8s) Cluster Creation Fails in Container Service Extension (CSE) 4.2.2

Products

• VMware Cloud Director

Issue/Introduction

• Creation of Kubernetes clusters fails when deployed via Container Service Extension (CSE).

• Errors are observed in the cloud-final.err log during the deployment process:

Jun 24 20:32:03 test-cls-spo@CSE/cse-user: [x] Fatal, plugins discovery image signature verification failed. The `tanzu` CLI can not ensure the integrity of the plugins to be installed. To ignore this validation please append "projects.registry.vmware.com/tanzu_cli/plugins/plugin-inventory:latest" to the comma-separated list in the environment variable "TANZU_CLI_PLUGIN_DISCOVERY_IMAGE_SIGNATURE_VERIFICATION_SKIP_LIST".  This is NOT RECOMMENDED and could put your environment at risk!'

• The issue may present itself on environments that have been running stable for months without recent upgrades.

Environment

• VMware Cloud Director (vCD) 10.6.x

• Container Service Extension (CSE) 4.2.2 (including CSE plugin)

• Tanzu Kubernetes Grid (TKG) 2.5.0

Cause

This failure occurs due to known deployment compatibility issues with older versions of CSE (4.2.2) and TKG (2.5.0). The Tanzu CLI no longer trusts the signatures of the plugins which need to be installed.

Resolution

To resolve this issue, the environment must be upgraded to newer, supported versions of CSE and TKGm.

Follow these steps:

1. Upgrade CSE: Update VMware Cloud Director Container Service Extension to version 4.2.4. Download is available here.

2. Upgrade TKGm: Update Tanzu Kubernetes Grid to version 2.5.4. Downloads are available here.

3. Deploy Updated OVAs: Ensure you are using the latest compatible Ubuntu K8s Tanzu OVA files for the deployment (e.g., ubuntu-2204-kube-v1.28.15+vmware.7-tkg.1-9f7eb2.ova).
   
   Note on Entitlements: If you experience a blank page or permission error when attempting to download TKGm 2.5.4 from the Broadcom portal, this typically indicates an entitlement or contract mismatch. In these instances, please contact the Broadcom entitlement team by opening a new GCA (Global Customer Assistance) case to verify your account permissions.