VKS Supervisor deactivation fails with the error "Avi client not initialized."

Products

VMware vSphere Kubernetes Service Tanzu Kubernetes Runtime VMware Avi Load Balancer

Issue/Introduction

When attempting to delete or deactivate a vSphere Kubernetes Service (VKS) Supervisor Cluster, the process gets stuck removing because the deletion of Avi objects from NSX Manager fails. This failure causes the cleanup request to fail, preventing the VKS Supervisor Cluster from being deleted successfully.

Logs similar to the below can be found in the /var/log/vmware/wcp/wcpsvc.log in vCenter:

info wcp [cleanup/vpc.go:120] [opID=XXXXXXX] Running Avi cleanup on host '<HOST_IP>' with user '<USER>' for cluster '<CLUSTER_ID>'
error wcp [cleanup/vpc.go:72] [opID=XXXXXXX] Error cleaning Avi resources: cleanup failed cluster=<CLUSTER_NAME>: error cleaning Avi resources for cluster '<CLUSTER_ID>': Avi client not initialized
error wcp [kubelifecycle/controller.go:2531] [opID=XXXXXXX] Teardown of external appliance resources failed. Err: error cleaning NSX resources for Supervisor '<SUPERVISOR_ID>': failed to perform NSX cleanup for Supervisor '<SUPERVISOR_ID>': NSX cleanup failed for Supervisor '<SUPERVISOR_ID>': cleanup failed cluster=<CLUSTER_NAME>: error cleaning Avi resources for cluster '<CLUSTER_ID>': Avi client not initialized
warning wcp [kubelifecycle/controller.go:478] [opID=XXXXXXX] Unable to disable cluster '<CLUSTER_NAME>' because of reason [FailedWithSystemError]. Err: error cleaning NSX resources for Supervisor '<SUPERVISOR_ID>': failed to perform NSX cleanup for Supervisor '<SUPERVISOR_ID>': NSX cleanup failed for Supervisor '<SUPERVISOR_ID>': cleanup failed cluster=<CLUSTER_NAME>: error cleaning Avi resources for cluster '<CLUSTER_ID>': Avi client not initialized

Environment

VKS Supervisor

vSphere Supervisor

vSphere Cloud Foundation (VCF) 9

Cause

The issue occurs when the Avi resources associated with the Supervisor Cluster cannot be completely removed. The failure is caused by an unsuccessful AVI cleanup process.

Resolution

To resolve the issue, perform the following steps to unblock the Supervisor Cluster deactivation:

Uncheck the "Onboard Avi into NSX" by navigating to Avi ALB > Infrastructure > Clouds. Click Edit NSX Cloud

Deregister the Avi Controller from the NSX Manager

Step1: Verify the Registration and check the flag "managed_by" LCM or VCF

curl -k -u 'admin' \
--location --request GET \
'https://<nsx_mgr_ip>/policy/api/v1/infra/sites/default/enforcement-points/alb-endpoint'

Step2: Removing the Integration

curl -k --location -u 'admin' \
--request DELETE \
'https://<nsx_mgr_ip>/policy/api/v1/infra/alb-onboarding-workflow/<managed_by-flag-value>'

Reference Article
SSH to Avi Load Balancer Controller Node, Get into the "Shell" and Delete the users 1. nsxt-alb, nsxt-ako, nsxt-lcm from CLI
Restart the WCP service. The resource cleanup process should now complete successfully, allowing the Supervisor Cluster to deactivate.
```
# vmon-cli -r wcp
```
Once the Supervisor Cluster deletion is complete, re-register the Avi Controller with the NSX Manager by navigating to Avi ALB UI > Infrastructure > Cloud > Click Edit NSX Cloud enable the check the "Onboard Avi into NSX"

Note: After 5min Avi ALB auto register / on-board with NSX manager as per the cloud reconcile workflow.

TechDoc Config Reference
Verify the Avi ALB registration - KB Article

Additional Information

If the Avi Controller is deployed using NSX Manager, it must be undeployed through the NSX Manager UI — this will result in the deletion of the Avi Controller VM. Avi can then be redeployed to obtain a Supervisor with Avi Load Balancer.

However, if Avi was deployed as a standalone VM, it must be explicitly deregistered from NSX using the following API call:

To register the Avi Controller back with NSX, use the following API call:

Payload: