Enable the Antrea-NSX Adapter for an existing TKG Service Cluster

Products

• VMware NSX
• VMware vSphere ESXi
• VMware vSphere with Tanzu

Issue/Introduction

Enable or disable Antrea-NSX integration for an existing TKG cluster, for vSphere 8.0 U3.

Environment

vSphere 8 U3 (8.0.3) or later

NSX 4.1 or later

TKG Service 3.0 or later

Tanzu Kubernetes release v1.28.x for vSphere 8.x or later

Cause

This is not officially supported.

Impact:

customers had to configure AntreaConfig.spec.antreaNSX.enable before deploying a TKG cluster. The configuration cannot be changed after the TKG cluster is deployed.

 

Resolution

Workaround:

Enabling/disabling Antrea-NSX after the cluster is deployed is an experimental feature.

Set spec.antreaNSX.enable as true or false by following command:

kubectl edit antreaconfig -n ${CLUSTER_NAMESPACE} ${CLUSTER_NAMES}-antrea-package

Suppose the cluster name is "my-cluster", and namespace name is "my-ns", an example command is below.

kubectl edit antreaconfig -n my-nsx my-cluster-antrea-package

An editor will be started, and customers can use the editor to change the configuration. After saving and exiting the editor, the changes will be submitted automatically. Then the Antrea-NSX Adapter will be deployed once spec.antreaNSX.enabled set as true. But when customers want to disable this feature, it is necessary to set spec.antreaNSX.enabled set as first, and run following commandlines. Here we take the cluster guestcluster4 in namespace gctest as an example.

# kubectl get cluster -ngctest NAME            CLUSTERCLASS             PHASE         AGE   VERSION guestcluster4   tanzukubernetescluster   Provisioned   21h   v1.28.8+vmware.1-fips.1   # kubectl get nsxserviceaccounts.nsx.vmware.com -ngctest NAME                   AGE guestcluster4-antrea   20h   # kubectl get providerserviceaccounts.vmware.infrastructure.cluster.x-k8s.io -ngctest NAME                                 VSPHERECLUSTER        TARGETNAMESPACE                      TARGETSECRETNAME                AGE guestcluster4-rwg9z-antrea           guestcluster4-rwg9z   vmware-system-antrea                 supervisor-cred                 20h guestcluster4-rwg9z-ccm              guestcluster4-rwg9z   vmware-system-cloud-provider         cloud-provider-creds            21h guestcluster4-rwg9z-pvbackupdriver   guestcluster4-rwg9z   velero-vsphere-plugin-backupdriver   pvbackupdriver-provider-creds   21h guestcluster4-rwg9z-pvcsi            guestcluster4-rwg9z   vmware-system-csi                    pvcsi-provider-creds            21h

Delete the following two crs, then the cluster is de-registered from NSX manager:

kubectl delete nsxserviceaccounts.nsx.vmware.com -ngctest guestcluster4-antrea kubectl delete providerserviceaccounts.vmware.infrastructure.cluster.x-k8s.io -ngctest guestcluster4-rwg9z-antrea