Workload cluster upgrade fails with the error: unknown command "cluster" for "tanzu"

Products

• VMware Telco Cloud Automation

Issue/Introduction

Workload cluster upgrade fails with the error:

Cluster Upgrade failed with Error:Error: unknown command "cluster" for "tanzu"

You also observe that some plugins are not present when you run tanzu plugin list as the user 'root' on the TCA-CP:

Standalone Plugins
  NAME                DESCRIPTION                                                        TARGET      DISCOVERY  VERSION  STATUS
  isolated-cluster    isolated-cluster operations                                                    default    v0.28.1  installed
  login               Login to the platform                                                          default    v0.28.1  installed
  pinniped-auth       Pinniped authentication operations (usually not directly invoked)              default    v0.28.1  installed
  management-cluster  Kubernetes management-cluster operations                           kubernetes  default    v0.28.1  installed
  package             Tanzu package management                                           kubernetes  default    v0.28.1  installed
  secret              Tanzu secret management                                            kubernetes  default    v0.28.1  installed
  telemetry           Configure cluster-wide telemetry settings                          kubernetes  default    v0.28.1  installed

*** MISSING ***

Plugins from Context:  <CONTEXT>
  NAME                DESCRIPTION                           TARGET      VERSION  STATUS
  cluster             Kubernetes cluster operations         kubernetes  v0.28.1  installed
  feature             Operate on features and featuregates  kubernetes  v0.28.1  installed
  kubernetes-release  Kubernetes release operations         kubernetes  v0.28.1  installed

Environment

Telco Cloud Automation

Cause

This issue can be caused by an incorrect endpoint being configured for the TKG_CUSTOM_IMAGE_REPOSITORY variable. The necessary components fail to download, leaving the solution in an inconsistent state.

Resolution

• To validate this is the issue, check the value for the TKG_CUSTOM_IMAGE_REPOSITORY variable on the TCA-CP:

su root
cat ~/.config/tanzu/config.yaml | grep TKG_CUSTOM_IMAGE_REPOSITORY

An example of a correct configuration:
TKG_CUSTOM_IMAGE_REPOSITORY: registry.example.com/yourproject

An example of an incorrect configuration:
TKG_CUSTOM_IMAGE_REPOSITORY: registry.example.com/yourproject/packages/management

 

• To resolve this issue, we need to provide correct values to the environment variables to point to the correct registry endpoint.

export TKG_CUSTOM_IMAGE_REPOSITORY: harbor.registry.com/yourproject

• To avoid any certificate trust errors from causing failure, also provide a base64 encoded version of the registry's TLS certificate.

export TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE="<BASE64_CERTIFICATE>"

• With the correct repository endpoint configured, apply the changes:

tanzu management-cluster upgrade
tanzu plugin clean
tanzu plugin sync
tanzu init

• Confirm the plugins are now available:

tanzu plugin list

• Retry the upgrade.

Additional Information

It's possible other plugins are the cause of this error. These steps can also resolve the scenario where other plugins are missing.