Error: "ctr: failed to resolve reference "projects.packages.broadcom.com/vmware-cloud-director/csi-crs-airgapped:<version>": projects.packages.broadcom.com/vmware-cloud-director/csi-crs-airgapped:<version>: not found" when creating a Kubernetes cluster

Products

• VMware Cloud Director

Issue/Introduction

• Attempting to create a Kubernetes cluster in Cloud Director using Container Service Extension fails.
• The EPHEMERAL-TEMP-VM is created and powered on but no Control Plane or Worker nodes are deployed during cluster Kubernetes creation.
• The Kubernetes cluster creation fails with an error in the Cloud Director UI of the form:

[error while bootstrapping the machine [<cluster-name>/EPHEMERAL-TEMP-VM]; unable to wait for post customization phase [guestinfo.cloudinit.kind.binary.install.status] : [invalid postcustomization phase: [failed] for key [guestinfo.cloudinit.kind.binary.install.status] for vm [EPHEMERAL-TEMP-VM] due to :[ctr image pull projects.registry.vmware.com/$1/$2]]] during cluster creation

• The /var/log/cloud-final.err logs on the EPHEMERAL-TEMP-VM show a failure to pull the csi-crs-airgapped image:

<DATE> <user>@System/<username>: + ctr image pull projects.packages.broadcom.com/vmware-cloud-director/csi-crs-airgapped:<version>
...
<DATE> 12 10:31:01 <user>@System/<username>: time="<DATE>" level=info msg="trying next host - response was http.StatusNotFound" host=projects.packages.broadcom.com
<DATE> 12 10:31:01 <user>@System/<username>: ctr: failed to resolve reference "projects.packages.broadcom.com/vmware-cloud-director/csi-crs-airgapped:<version>": projects.packages.broadcom.com/vmware-cloud-director/csi-crs-airgapped:<version>: not found

Environment

VMware Cloud Director

VMware Cloud Director Container Service Extension

Cause

This error occurs if an invalid Container Storage Interface (CSI) version has been set in the Cloud Director Provider portal, Kubernetes Container Clusters plug-in, CSE Management, Server Details.

Resolution

Update the CSE Server configuration in the Cloud Director Provider portal, Kubernetes Container Clusters plug-in, CSE Management, Server Details., set the Container Storage Interface (CSI) version to a valid release, and restart the CSE Server.
Details on this process are outlined in the documentation, Update Server Configuration.

Details of the available Container Storage Interface (CSI) versions can be found on the component's GitHub page, Container Storage Interface (CSI) driver for VMware Cloud Director Named Independent Disks.