Antrea NSX Interworking Adapter migration from a manual deployment to auto-deployment in an existing VKS Cluster
search cancel

Antrea NSX Interworking Adapter migration from a manual deployment to auto-deployment in an existing VKS Cluster

book

Article ID: 445655

calendar_today

Updated On:

Products

VMware NSX VMware Container Networking with Antrea VMware vSphere Kubernetes Service

Issue/Introduction

This KB outlines steps to migrate from a manually registered Antrea NSX Interworking adapter (provisioned via manual manifests or ArgoCD) to a VKr-managed auto-registered adapter.

Environment

VMware NSX
VMware Antrea
VMware vSphere Kubernetes Service

Resolution

Prerequisites

  1. Applicable only if there are no NSX Security Policies applied to already registered Antrea Container Cluster (VKS Cluster).
  2. Applicable only if there are no NSX Generic groups with Kubernetes or Antrea members for already registered Antrea Container Cluster (VKS Cluster).
  3. Ensure that all NSX IPs are reachable from Antrea Container Cluster (VKS Cluster) Nodes. If not, first Install NSX Management Proxy Supervisor Service for VCF < 9.0 or Install Supervisor Management Proxy for VCF >= 9.0
  4. If NSX has published FQDN, ensure that all NSX FQDNs are resolvable from Antrea Container Cluster (VKS Cluster) Nodes. If NSX FQDNs are not resolvable from Antrea Container Cluster (VKS Cluster) Nodes, follow below mentioned workaround to resolve NSX FQDNs from interworking Pod running in Antrea Container Cluster (VKS Cluster).
  5. How to check if your NSX has FQDNs and what are those FQDNs.

    1. On Supervisor Cluster, run following command to get nsx-ncp-config configmap.

      kubectl -n vmware-system-nsx get cm nsx-ncp-config -o yaml

    2. In the output of above configmap data, look for values corresponding to key "nsx_api_managers".
    3. In above output, if values corresponding to key "nsx_api_managers" has NSX IPs only, then check NSX FQDNs using following command on Supervisor cluster.

      curl -k -u admin:${NSX_ADMIN_PASSWORD} https://${NSX_IP}/api/v1/cluster/nodes | grep fqdn

  6. If an Antrea Container Cluster (VKS cluster) is registered manually to NSX, its name will change in NSX following auto-registration. The new naming convention will be:

    "<supervisor-id>-<vsphere-namespace-of vks-cluster>-<vks-cluster-name>-antrea"

 

Workaround to resolve NSX FQDNs from interworking Pod

  1. Applicable only if NSX FQDNs are not resolvable from Antrea Container Cluster (VKS Cluster) Nodes. This workaround can be done before Auto register Antrea-NSX for a VKS Cluster step.
  2. Create the following secret in VKS cluster. NOTE: You MUST append all FQDN-to-IP mappings of your NSX to hostAliases list below before applying.

    host-alias-overlay.yml

    apiVersion: v1
    kind: Secret
    metadata:
      name: host-alias-overlay
      namespace: vmware-system-tkg
    stringData:
      add-host-aliases.yml: |
        #@ load("@ytt:overlay", "overlay")

        #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata": {"name": "interworking"}}), expects="0+"
        ---
        spec:
          template:
            spec:
              #@overlay/match missing_ok=True
              hostAliases:
              - ip: "198.51.100.21" # <--- Change to your NSX IP
                hostnames:
                - "web-node01.internal.example.com" # <--- Change to your NSX FQDN
              - ip: "198.51.100.22" # <--- Change to your second NSX IP
                hostnames:
                - "web-node02.internal.example.com" # <--- Change to your second NSX FQDN
              - ip: "198.51.100.23" # <--- Change to your third NSX IP
                hostnames:
                - "web-node03.internal.example.com" # <--- Change to your third NSX FQDN

  3. Update PackageInstall(pkgi) CR  annotation in VKS Cluster to use above created host-alias-overlay secret. Example - If your Antrea Container Cluster (VKS Cluster) name is "cluster-default", then its corresponding PackageInstall(pkgi) CR name is "cluster-default-antrea".

    Edit pkgi annotation

    kubectl -n vmware-system-tkg edit pkgi cluster-default-antrea

    Example pkgi with annotation

    apiVersion: packaging.carvel.dev/v1alpha1
    kind: PackageInstall
    metadata:
      annotations:
        ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: host-alias-overlay # <--- Add this annotation
        tkg.tanzu.vmware.com/cluster-name: cluster-default
        tkg.tanzu.vmware.com/cluster-namespace: antrea-test
      creationTimestamp: "2026-06-12T11:40:24Z"
      finalizers:
      - finalizers.packageinstall.packaging.carvel.dev/delete
      generation: 1
      name: cluster-default-antrea
      namespace: vmware-system-tkg
      resourceVersion: "580017"
      uid:######-####-####-####-########
    spec:
      packageRef:
        refName: antrea.tanzu.vmware.com
        versionSelection:
          constraints: 2.4.4+vmware.1-tkg.1
          prereleases: {}
      serviceAccountName: tanzu-cluster-bootstrap-sa
      syncPeriod: 10m0s
      values:
      - secretRef:
          name: cluster-default-antrea-data-values
    status:
      conditions:
      - status: "True"
        type: ReconcileSucceeded
      friendlyDescription: Reconcile succeeded
      lastAttemptedVersion: 2.4.4+vmware.1-tkg.1
      observedGeneration: 1
      version: 2.4.4+vmware.1-tkg.1



  4. Verify that reconcile is successful for edited PackageInstall(pkgi) in VKS Cluster.

    root@######## [ ~ ]# kubectl -n vmware-system-tkg get pkgi cluster-default-antrea
    NAME                     PACKAGE NAME              PACKAGE VERSION        DESCRIPTION           AGE   PAUSED
    cluster-default-antrea   antrea.tanzu.vmware.com   2.4.4+vmware.1-tkg.1   Reconcile succeeded   11d

  5. To verify VKS Cluster after doing auto registration for Antrea-NSX.

    1. Resolve NSX FQDN from interworking pod. Replace interworking pod name and NSX FQDN in below example. Check the output carefully for resolved NSX IP, connectivity to that resolved NSX IP may fail if workload and management isolated but that is not a concern here. We just want that NSX FQDN should be resolvable to NSX IP.

      kubectl -n vmware-system-antrea exec -it interworking-5d8d9dd58c-t7zxj -- curl -kv https://web-node01.internal.example.com

    2. Interworking deployment in VKS cluster is updated with hostAliases.

      kubectl -n vmware-system-antrea get deploy interworking -oyaml
      ...
            dnsPolicy: ClusterFirst
            hostAliases:
            - hostnames:
              - web-node01.internal.example.com # <--- First NSX FQDN
              ip: 198.51.100.21 # <--- First NSX IP
            - hostnames:
              - web-node02.internal.example.com # <--- Second NSX FQDN
              ip: 198.51.100.22 # <--- Second NSX IP
            - hostnames:
              - web-node03.internal.example.com # <--- Third NSX FQDN
              ip: 198.51.100.23 # <--- Third NSX IP
            hostNetwork: true
      ...


    3. /etc/hosts file inside new interworking pod is updated with NSX FQDN. Replace interworking pod name in below example.

      /etc/hosts file from interworking pod

      kubectl -n vmware-system-antrea exec -it interworking-######## -- cat /etc/hosts
      Defaulted container "election-runner" out of: election-runner, mp-adapter, tn-proxy, ccp-adapter

      Kubernetes-managed hosts file (host network).

      ::1 ipv6-localhost ipv6-loopback
      127.0.0.1 localhost cluster-default-trzhj-jhbgt

      Entries added by HostAliases.

      198.51.100.21 web-node01.internal.example.com # <--- First NSX IP and FQDN
      198.51.100.22 web-node02.internal.example.com # <--- Second NSX IP and FQDN
      198.51.100.23 web-node03.internal.example.com # <--- Third NSX IP and FQDN




Deregister manually registered Antrea-NSX for a VKS Cluster

  1. If your Antrea Container Cluster (VKS Cluster) is already registered to NSX manually, follow doc to deregister your Antrea Container Cluster (VKS Cluster) from NSX.


Auto register Antrea-NSX for a VKS Cluster (VKr 1.34)

  1. Ensure that your Antrea Container Cluster (VKS Cluster) is not already registered to NSX(manually or auto). If manually registered, follow above steps to deregister first.
  2. For every VKS Cluster, there is corresponding AntreaConfig CR in Supervisor Cluster in the vsphere namespace same as Antrea Container Cluster (VKS Cluster). AntreaConfig CR name follows the convention "<vks-cluster-name>-antrea-package". Example - If your Antrea Container Cluster (VKS Cluster) name is "cluster-default", then its corresponding AntreaConfig CR name is "cluster-default-antrea-package"

    Get AntreaConfig in Supervisor Cluster

    kubectl -n <vsphere-namespace-of-vks-cluster> get antreaconfig

  3. To enable auto-registration, set spec.antreaNSX.enable: true in the AntreaConfig CR (corresponding to the VKS Cluster) in the Supervisor Cluster.
    Example - Assuming that in "antrea-test" vsphere namespace, there is Antrea Container Cluster (VKS Cluster) with name "cluster-default" and its corresponding AntreaConfig CR name is "cluster-default-antrea-package"

    Edit AntreaConfig CR

    kubectl -n antrea-test edit antreaconfig cluster-default-antrea-package

    Example AntreaConfig
    apiVersion: cni.tanzu.vmware.com/v1alpha1
    kind: AntreaConfig
    metadata:
      name: cluster-default-antrea-package
      namespace: antrea-test
    spec:
      antreaNSX:
        enable: true # <--- Set to true



Auto register Antrea-NSX for a VKS Cluster (VKr >= 1.35 + VKS >= 3.6)

  1. Same as step 1 above.
  2. For every VKS Cluster, there is corresponding AddonConfig CR in Supervisor Cluster in the vsphere namespace same as Antrea Container Cluster (VKS Cluster). AddonConfig CR name follows the convention "<vks-cluster-name>-antrea". Example - If your Antrea Container Cluster (VKS Cluster) name is "cluster-default", then its corresponding AddonConfig CR name is "cluster-default-antrea".

    Get AddonConfig in Supervisor Cluster

    kubectl -n <vsphere-namespace-of-vks-cluster> get addonconfig

  3. To enable auto-registration, set spec.values.antreaNSX.enable: true in the AddonConfig CR (corresponding to the VKS Cluster) in the Supervisor Cluster. Example - Assuming that in "antrea-test" vsphere namespace, there is Antrea Container Cluster (VKS Cluster) with name "cluster-default" and its corresponding AddonConfig CR name is "cluster-default-antrea".

    Edit AddonConfig CR

    kubectl -n antrea-test edit addonconfig cluster-default-antrea

    Example AddonConfig
    apiVersion: addons.kubernetes.vmware.com/v1alpha1
    kind: AddonConfig
    metadata:
      name: cluster-default-antrea
      namespace: antrea-test
    spec:
      addonConfigDefinitionRef:
        name: antrea.tanzu.vmware.com.2.5.1---vmware.2-tkg.1
        namespace: vmware-system-vks-public
      clusterName: cluster-default
      values:
        antreaNSX:
          enable: true # <--- Set to true
Powered by Wolken Software