Upgrade of cluster fails with error: "Upload NSX-LCP bundle operation failed" due to timeout

Products

• VMware NSX

Issue/Introduction

During an NSX upgrade, a cluster fails to upgrade.  The status error message for the failed upgrade is a long message resembling the following:

"Upload NSX-LCP bundle operation failed: com.vmware.nsx.management.ovfops.exception.CommandExecutionFailed: Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 179M 0 0 0 511k 0 501k 0:06:06 0:00:01 0:06:05 501k 0 179M 0 0 0 1343k 0 665k 0:04:36 0:00:02 0:04:34 665k 1 179M 0 0 1 2175k 0 720k 0:04:15 0:00:03 0:04:12 720k 1 179M 0 0 1 2943k 0 732k 0:04:10 0:00:04 0:04:06 732k 2 179M 0 0 2 3775k 0 752k 0:04:04 0:00:05 0:03:59 764k 2 179M 0 0 2 4543k 0 754k 0:04:03 0:00:06 0:03:57 806k 2 179M 0 0 2 5343k 0 761k 0:04:01 0:00:07 0:03:54 800k 3 179M 0 0 3 5767k 0 719k 0:04:15 0:00:08 0:04:07 718k 3 179M 0 0 3 6231k 0 690k 0:04:25 0:00:09 0:04:16 657k 3 179M 0 0 3 6695k 0 668k 0:04:34 0:00:10 0:04:24 584k 3 179M 0 0 3 7119k 0 646k 0:04:44 0:00:11 0:04:33 515k 4 179M 0 0 4 7583k 0 630k 0:04:51 0:00:12 0:04:39 447k 4 179M 0 0 4 8007k 0 615k 0:04:58 0:00:13 0:04:45 448k 4 179M 0 0 4 8471k 0 604k 0:05:04 0:00:14 0:04:50 448k 4 179M 0 0 4 8935k 0 594k 0:05:08 0:00:15 0:04:53 448k 5 179M 0 0 5 9359k 0 584k 0:05:14 0:00:16 0:04:58 448k 5 179M 0 0 5 9823k 0 577k 0:05:18 0:00:17 0:05:01 448k 5 179M 0 0 5 10.0M 0 570k 0:05:21 0:00:18 0:05:03 455k 5 179M 0 0 5 10.4M 0 563k 0:05:26 0:00:19 0:05:07 448k 6 179M 0 0 6 10.9M 0 558k 0:05:29 0:00:20 0:05:09 448k 6 179M 0 0 6 11.3M 0 551k 0:05:33 0:00:21 0:05:12 448k 6 179M 0 0 6 11.7M 0 547k 0:05:35 0:00:22 0:05:13 448k 6 179M 0 0 6 12.1M 0 542k 0:05:38 0:00:23 0:05:15 440k 7 179M 0 0 7 12.6M 0 539k 0:05:40 0:00:24 0:05:16 448k 7 179M 0 0 7 13.1M 0 536k 0:05:42 0:00:25 0:05:17 448k 7 179M 0 0 7 13.5M 0 531k 0:05:45 0:00:26 0:05:19 448k 7 179M 0 0 7 13.9M 0 529k 0:05:47 0:00:27 0:05:20 448k 8 179M 0 0 8 14.4M 0 527k 0:05:48 0:00:28 0:05:20 455k 8 179M 0 0 8 14.8M 0 523k

...

0:04:59 0:01:43 449k curl: (28) Operation timed out after 300000 milliseconds with 0 bytes received"

 

 

Environment

VMware NSX-T DataCenter
VMware NSX

Cause

The NSX-LCP bundle file copy takes more than 5 minutes to transfer to the ESXi host. 

Resolution

Investigate host network connectivity for slow performance and ensure all required ports are opened.  If the cluster that failed to upgrade is vLCM enabled, note the requirement for port 9087 between the NSX Managers and vCenter.

Workaround:

• Manually copy the LCP file from the NSX Manager node from /repository into the /tmp folder of the ESXi hosts within the affected cluster.  This can be achieved via WinSCP or other file transfer utilities.
  • File path for LCP file from root user SSH session NSX Manager:
    • /repository/<new-NSX-version>/HostComponents/<esx-version>/nsx-lcp-<new-nsx-version>-<esx-version>.zip
    • e.g. /repository/4.2.0.0.0.24105817/HostComponents/esx80/nsx-lcp-4.2.0.0.0.24105819-esx80.zip
• Copy the file to each host in the affected cluster, then run the following command on each of those hosts:
  • esxcli software component apply -n nsx-lcp-bundle  -d /tmp/nsx-lcp-<new-nsx-version>-<esx-version>.zip --no-sig-check
• Once done, the NSX cluster upgrade should go through successfully.