vSphere with Tanzu deactivation fails with the error "Avi client not initialized."

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When attempting to delete a Supervisor Cluster, the process gets stuck because the deletion of Avi objects from NSX Manager fails. This failure causes the cleanup request to fail, preventing the Supervisor Cluster from being deleted successfully.

/var/log/vmware/wcp/wcpsvc.log:

info   wcp [cleanup/vpc.go:120] [opID=XXXXXXX] Running Avi cleanup on host '<HOST_IP>' with user '<USER>' for cluster '<CLUSTER_ID>'
error  wcp [cleanup/vpc.go:72] [opID=XXXXXXX] Error cleaning Avi resources: cleanup failed cluster=<CLUSTER_NAME>: error cleaning Avi resources for cluster '<CLUSTER_ID>': Avi client not initialized
error  wcp [kubelifecycle/controller.go:2531] [opID=XXXXXXX] Teardown of external appliance resources failed. Err: error cleaning NSX resources for Supervisor '<SUPERVISOR_ID>': failed to perform NSX cleanup for Supervisor '<SUPERVISOR_ID>': NSX cleanup failed for Supervisor '<SUPERVISOR_ID>': cleanup failed cluster=<CLUSTER_NAME>: error cleaning Avi resources for cluster '<CLUSTER_ID>': Avi client not initialized
warning  wcp [kubelifecycle/controller.go:478] [opID=XXXXXXX] Unable to disable cluster '<CLUSTER_NAME>' because of reason [FailedWithSystemError]. Err: error cleaning NSX resources for Supervisor '<SUPERVISOR_ID>': failed to perform NSX cleanup for Supervisor '<SUPERVISOR_ID>': NSX cleanup failed for Supervisor '<SUPERVISOR_ID>': cleanup failed cluster=<CLUSTER_NAME>: error cleaning Avi resources for cluster '<CLUSTER_ID>': Avi client not initialized

Environment

VMware vSphere with Tanzu

Cause

The issue occurs when the Avi resources associated with the Supervisor Cluster cannot be completely removed. The failure is caused by an unsuccessful AVI cleanup process.

Resolution

To resolve the issue, perform the following steps to unblock the Supervisor Cluster deactivation:

• Deregister the Avi Controller from the NSX Manager.

• Restart the WCP service. The resource cleanup process should now complete successfully, allowing the Supervisor Cluster to deactivate.

                # vmon-cli -r wcp

• Once the Supervisor Cluster deletion is complete, re-register the Avi Controller with the NSX Manager.

Additional Information

If the Avi Controller is deployed using NSX Manager, it must be undeployed through the NSX Manager UI — this will result in the deletion of the Avi Controller VM. Avi can then be redeployed to obtain a Supervisor with Avi Load Balancer.

However, if Avi was deployed as a standalone VM, it must be explicitly deregistered from NSX using the following API call:

https://<nsx-mgr-ip>/policy/api/v1/infra/alb-onboarding-workflow/LCM

 

To register the Avi Controller back with NSX, use the following API call:

 

https://<nsx-mgr-ip>/policy/api/v1/infra/alb-onboarding-workflow

Payload:

 

{
  "owned_by": "LCM",
  "cluster_ip": "<avi-ctrl-ip>",
  "infra_admin_username": "<avi-admin-user>",
  "infra_admin_password": "<avi-admin-password>"
}