Unable to create or upgrade workload clusters post upgrading the VKS Service to 3.5.0

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Unable to create a new workload cluster or upgrade one after the VKS service is upgraded to version 3.5.0.
  
  
• The describe output of the cluster looks like below.
  
  Status:
  Conditions:
    Last Transition Time:  YYYY-MM-DDT<time:timezone>
    Message:               error reconciling the Cluster topology: failed to mark "AfterControlPlaneInitialized" hook(s) as pending: failed to patch Cluster: admission webhook "capi.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: no kind "Cluster" is registered for version "cluster.x-k8s.io/v1beta2" in scheme "pkg/runtime/scheme.go:100"
    Reason:                TopologyReconcileFailed
    Severity:              Error
    Status:                False
    Type:                  TopologyReconciled
  Control Plane Ready:     false
  Infrastructure Ready:    false
  v1beta2:
    Conditions:
      Last Transition Time:  YYYY-MM-DDT<time:timezone>
      Message:               error reconciling the Cluster topology: failed to mark "AfterControlPlaneInitialized" hook(s) as pending: failed to patch Cluster: admission webhook "capi.mutating.tanzukubernetescluster.run.tanzu.vmware.com" denied the request: no kind "Cluster" is registered for version "cluster.x-k8s.io/v1beta2" in scheme "pkg/runtime/scheme.go:100"
      Observed Generation:   3
      Reason:                ReconcileFailed
      Status:                False
      Type:                  TopologyReconciled
Events:                      <none>

  
  
• Some other symptoms include VKS upgrade pre-check warnings regarding unsigned bundles, system failure status following an upgrade attempt and failure of WCP to reconcile or install VKS packages.
  
  
• This issue occurs only on Async SV OR vSphere 9.0.1 or greater with VKS 3.5.0 with an unsigned bundle.

Environment

VMware vSphere Kubernetes Service

Cause

A security enforcement change introduced in Async Supervisor (SV) requires all new VKS bundles (starting with version 3.5.0) to possess a valid signature for installation. While earlier versions were whitelisted for backward compatibility, version 3.5.0 and above are subject to strict signature verification to ensure supply chain integrity.

Resolution

To address the installation or upgrade failure,

• The VKS bundle must be pushed to the registry with a valid signature.
• Verify the VKS Version: Confirm if the target version is 3.5.0 or greater.
• Review Documentation: Consult the VKS product documentation for the specific requirements on adding signatures during the bundle push process.
• Sign the Bundle: Ensure the signing utility is used to apply the Broadcom/VMware signature before pushing the bundle to the Supervisor registry.
• Address the pre-check warnings: If the VKS upgrade pre-check displays a warning regarding an unsigned bundle, do not proceed with the upgrade until the bundle has been correctly signed and re-pushed.
• Re-attempt Installation: Once the signed bundle is available in the registry, trigger the WCP reconciliation or upgrade again.