NSX upgrade fails on host stage with message stating, "Failed to execute localcli cmd: %s, rc: %d, out: %s err: %s"
search cancel

NSX upgrade fails on host stage with message stating, "Failed to execute localcli cmd: %s, rc: %d, out: %s err: %s"

book

Article ID: 429730

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • An NSX upgrade fails on the stage of upgrading the ESXi hosts. A message similar to the following is displayed:

    Upgrade failed for ####### host. Please reboot the host immediately to discard the unfinished update.
    Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host a56abd3e-####-####-####-e56649bdf942 with error : [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_<version>.<build>: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1334, in <module> if not nsxesxutils.IsDualLoading(True): File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 529, in IsDualLoading nsx_version, nsx_esx_vib = getNSXFromVersionUpgrade() File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 428, in getNSXFromVersionUpgrade vib_list = localcli('software vib list') File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 276, in localcli raise Exception('Failed to execute localcli cmd: %s, rc: %d, out: %s err: %s' % Exception: Failed to execute localcli cmd: software vib list, rc: 1, out: err: Killed Errors: Got no data from process. Command "/usr/lib/vmware/esxcli-software vib.list " еxited with error code: 1 It is not safe to continue. Please reboot the host immediately to discard the unfinished update. cause = ('nsx-lcp-bundle(<version>.<build>)', 'nsx-datapath-dl', 'Error in running [/etc/init.d/nsx-datapath-dl start upgrade]:\nReturn code: 1\nOutput: start upgrade begin\nException:\nTraceback (most recent call last):\n File "/etc/init.d/nsx-datapath-dl", line 1334, in <module>\n if not nsxesxutils.IsDualLoading(True):\n File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 529, in IsDualLoading\n nsx_version, nsx_esx_vib = getNSXFromVersionUpgrade()\n File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 428, in getNSXFromVersionUpgrade\n vib_list = localcli(\'software vib list\')\n File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 276, in localcli\n raise Exception(\'Failed to execute localcli cmd: %s, rc: %d, out: %s err: %s\' %\nException: Failed to execute localcli cmd: software vib list, rc: 1, out: err: Killed\nErrors: \nGot no data from process.\nCommand "/usr/lib/vmware/esxcli-software vib.list "\nеxited with error code: 1\n\n') vibs = ['VMware_bootbank_nsx-esx-datapath_<version>.<build>'] Please refer to the log file for more details..

  • Messages similar to the following are present in the /var/run/log/vobd.log on the affected ESXi host:

    2026-01-30T19:43:30.898Z In(14) vobd[2097867]:  [GenericCorrelator] 25413059330302us: [vob.user.esximage.install.error] Could not install image profile: VMware_bootbank_nsx-esx-datapath_<version>.<build>: VMware_bootbank_nsx-esx-datapath_<version>.<build>: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]:
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Return code: 1
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Output: start upgrade begin
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Exception:
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Traceback (most recent call last):
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/etc/init.d/nsx-datapath-dl", line 1354, in <module>
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     DualLoadUpgrade()
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/etc/init.d/nsx-datapath-dl", line 1126, in DualLoadUpgrade
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     PreUpgrade()
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/etc/init.d/nsx-datapath-dl", line 949, in PreUpgrade
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     UnloadNonDLModules()
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/etc/init.d/nsx-datapath-dl", line 160, in UnloadNonDLModules
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     if IPfixUtil.UnloadIPfixModule(sw_with_ipfix_list, log) == -1:
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 837, in UnloadIPfixModule
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     cls.from_nsx_version, _ = getNSXFromVersionUpgrade()
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 428, in getNSXFromVersionUpgrade
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     vib_list = localcli('software vib list')
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:   File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 276, in localcli
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:     raise Exception('Failed to execute localcli cmd: %s, rc: %d, out: %s err: %s' %
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Exception: Failed to execute localcli cmd: software vib list, rc: 1, out:  err: Killed
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Errors:
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Got no data from process.
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: Command "/usr/lib/vmware/esxcli-software vib.list "
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: еxited with error code: 1
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]:
    2026-01-30T19:43:30.898Z In(14)[+] vobd[2097867]: It is not safe to continue. Please reboot the host immediately to discard the unfinished update.

  • Messages similar to the following are present in the /var/run/log/vmkernel log on the affected ESXi host:

    2026-01-30T19:30:28.900Z In(182) vmkernel: cpu72:329759453)Admission failure in path: host/vim/vmvisor/esximage:sh.329759453
    2026-01-30T19:30:28.900Z In(182) vmkernel: cpu72:329759453)sh.329759453 (2388841033) requires 306528 KB, asked 306528 KB from esximage (711) which has 623008 KB occupied and 298592 KB available.
    2026-01-30T19:30:28.900Z Wa(180) vmkwarning: cpu72:329759453)WARNING: UserParam: 1519: sh: could not SetAlloc on container(2388841033) -- 76800 pages: Admission check failed for memory resource
    2026-01-30T19:30:28.900Z Wa(180) vmkwarning: cpu72:329759453)WARNING: LinuxFileDesc: 4670: sh: Unrecoverable exec failure: Failure during exec while original state already lost

Environment

VMware NSX

Cause

This issue can occur when the esxcli (localcli) command requests more memory than it is allowed (900MB by default). 

Resolution

Once the issue has occurred, the only resolution is to reboot the affected host.

This issue can be worked around preemptively by increasing the memory limit for the esxcli command via the following commands:

/usr/lib/vmware/rp/bin/configRP getRPMemMaxSize host/vim/vmvisor/esximage 

You should get 900 back as a result.

/usr/lib/vmware/rp/bin/configRP increaseRPMemMaxSize host/vim/vmvisor/esximage 100

This increases the value by 100, to 1000.

/usr/lib/vmware/rp/bin/configRP getRPMemMaxSize host/vim/vmvisor/esximage

You should now get 1000 back as a result.

This workaround is only temporary and will be reverted the next time the host is rebooted.

Powered by Wolken Software