NAPP pane in NSX UI shows message "Server version <version> and client version <version> are incompatible. Please upload Kubernetes Tools to resolve." (or) k8tool upload to NSX manager failed on NAPP-Automation Appliance UI.

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

Symptoms:

1) NSX Application Platform pane in NSX UI shows message "Server version <version> and client version <version> are incompatible. Please upload Kubernetes Tools to resolve."
Running command /usr/bin/kubectl version on NSX Manager nodes as root user shows that not all Manager nodes have a Kubernetes Tools version compatible with the Server version.

(or)

2) Incase following error " k8tool upload to NSX manager failed" observed during NAPP deployment through NAPP Automation Appliance.




Following NAPP error logs observed under NSX manager - /var/log/proton/nsxapi.log
{"function":"CallAPI","level":"debug","msg":"Request: https://nfrnsx-vip.mgmt.local/policy/api/v1/infra/sites/default/napp/deployment/registry","time":"2024-10-07T12:28:41Z"}
{"function":"UpdateRegistry","level":"debug","msg":"Registry updated","time":"2024-10-07T12:28:41Z"}
{"function":"Deploy","level":"debug","msg":"Deployment","time":"2024-10-07T12:28:41Z"}
{"function":"CallAPI","level":"debug","msg":"Request: https://nfrnsx-vip.mgmt.local/policy/api/v1/infra/sites/default/napp/deployment/platform","time":"2024-10-07T12:28:41Z"}
{"function":"Deploy","level":"debug","msg":"Deployment started. Follow progress on the UI...","time":"2024-10-07T12:28:43Z"}
{"function":"GetKubeInfo","level":"debug","msg":"Retrieve Kubeconfig information","time":"2024-10-07T12:28:44Z"}
{"function":"CallAPI","level":"debug","msg":"Request: https://nfrnsx-vip.mgmt.local/policy/api/v1/infra/sites/default/napp/deployment/kubeconfig","time":"2024-10-07T12:28:44Z"}
{"function":"GetKubeInfo","level":"debug","msg":"Kubeconfig information retrieved","time":"2024-10-07T12:28:44Z"}
{"function":"UploadKubernetesTool","level":"warning","msg":"Bundle ID is: 1728304125713","time":"2024-10-07T12:28:46Z"}
{"function":"GetUploadBundleStatus","level":"debug","msg":"Get Upload Status","time":"2024-10-07T12:28:46Z"}
{"function":"CallAPI","level":"debug","msg":"Request: https://nfrnsx-vip.mgmt.local/policy/api/v1/infra/sites/default/napp/deployment/tools/upload/status/1728304125713","time":"2024-10-07T12:28:46Z"}
{"function":"GetUploadBundleStatus","level":"debug","msg":"Upload status retrieved","time":"2024-10-07T12:28:46Z"}
{"function":"GetUploadBundleStatus","level":"debug","msg":"Get Upload Status","time":"2024-10-07T12:28:49Z"}
{"function":"CallAPI","level":"debug","msg":"Request: https://nfrnsx-vip.mgmt.local/policy/api/v1/infra/sites/default/napp/deployment/tools/upload/status/1728304125713","time":"2024-10-07T12:28:49Z"}
{"function":"GetUploadBundleStatus","level":"debug","msg":"Upload status retrieved","time":"2024-10-07T12:28:49Z"}
{"function":"UploadKubernetesTool","level":"warning","msg":"k8s tools upload to NSX Manager failed - inspect/report with logs or support bundle from NSX Manager","time":"2024-10-07T12:28:49Z"}
{"function":"RunNappHandler","level":"info","msg":"NAPP deployment is completed without any error.","time":"2024-10-07T12:28:49Z"

Cause

The difference between the minor version number of the Kubernetes Tools on the NSX Manager appliance and on the Supervisor Kubernetes cluster is more than one. For example, if the version installed on the NSX Managers is 1.18 and Supervisor cluster uses 1.19, that is compatible. But if one of them uses version 1.18 and the other uses 1.20, the alert will be thrown.

Resolution

Kubernetes Tools is the Client version which runs on the NSX Managers. The Kubernetes Version in the NAPP pane is the Server version.

1) Go to Downloads page for the NSX Manager version and download a Kubernetes-tools bundle that is compatible with the Kubernetes Version in the NSX Application Platform pane (the Server version).

2) Then, click on Go to Configuration in the NSX Application Platform pane, Upload the new Kubernetes Tools file, and continue to update the configuration.

If you run into a similar error, as given below, please follow the workaround provided. 

" Server version 1.23.8 and client version 1.23.3 are incompatible. Please upload Kubernetes Tools to resolve."

1. Upload k8s tools of server version (1.23 in your case) on all managers. Please use NSX manager UI to upload tools (do not install tools manually).
2. Run below command on any one manager to cleanup tools sync status.

/opt/vmware/bin/corfu_tool_runner.py --tool corfu-editor -n nsx -t CloudNativeBundleSyncStatus -o clearTable

Note : Use NSX Application Platform under the NSX-UI for NAPP deployment ( and not NAPP Automation deployment wizard )

 

To download Kubernetes tools , you can use the below direct link :

https://support.broadcom.com/group/ecx/productfiles?displayGroup=VMware%20Firewall%20-%20NSX%20Distributed%20Firewall%20with%20Advanced%20Threat%20Protection&release=4.X&os=&servicePk=202456&language=EN&groupId=204048&viewGroup=true

 

Additional Information

Similar Resolution/Workaround

If you face the below error during upload of Kubernetes tools :

"Error: Payload size is: 26,191,531. It exceeds the max supported size of: 23,068,672 bytes. (Error code: 510)"

Note : NSXT 4.1.2/ 4.1.2.1 GA will have the issue 

Solution is to either use K8s version 1.23.x with NSX 4.1.2.1 or upgrade NSX to 4.1.2.3 where this has been fixed.

After obtaining the kubernetes file :

1. Untar downloaded file using command:

tar -xvf kubernetes-tools-... (complete file name)

2. Untar using above command will result into one more tar file and a sig file. Untar the tar file. 

tar -xvf tar -xvf kubernetes-debs-... (complete file name)

3. Above command will give 2 debian files for helm and kubectl. Install these two debian files on all NSX manager nodes.

dpkg -i <debian_name>

4. Start installing NAPP from NSX UI.

++++++