Failed to apply Component: an error occurred while enabling service nsx-datapath
search cancel

Failed to apply Component: an error occurred while enabling service nsx-datapath

book

Article ID: 407934

calendar_today

Updated On:

Products

VMware vCenter Server VMware Cloud Foundation

Issue/Introduction

  • Unable to install nsx vibs on the ESXi host using vLCM.
  • /var/run/log/lifecycle.log on ESXi host reports below : 
    • YYYY-MM-DDTHH:MM:SSZ Db(15) lifecycle[2733219]: HostImage:1454 installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_4.1.1.0.0-8.0.22224315: VMware_bootbank_nsx-esx-datapath_4.1.1.0.0-8.0.22224315: Error in running [/etc/init.d/nsx-datapath start install]:
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]: Return code: 1
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]: Output: start install begin
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]: Exception:
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]:   File "/etc/init.d/nsx-datapath", line 2000, in <module>
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]:     checkLKMs()
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]:   File "/etc/init.d/nsx-datapath", line 1631, in checkLKMs
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]:     raise Exception(msg)
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]: Exception: Failed to load kernel modules: nsxt-ipfix-22224315, nsxt-dvfilterUser-22224315, nsxt-core-22224315, nsxt-vdrb-22224315, nsxt-livetrace-22224315, nsxt-switch-security-22224315, nsxt-fc-22224315, nsxt-spf-22224315, nsxt-vswitch-22224315, nsxt-vsip-22224315, nsxt-ens-22224315, nsxt-vdl2-22224315, nsxt-traceflow-22224315, nsxt-latency-22224315, nsxt-obsrv-22224315, nsxt-bfd-22224315, nsxt-kcp-22224315
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]:
YYYY-MM-DDTHH:MM:SSZ Db(15)[+] lifecycle[2733219]: It is not safe to continue. Please reboot the host immediately to discard the unfinished update.. Clean up the installation.
  • /var/run/log/vmkernel.log:
    • YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.2733219:python.2733760:uw.2733760
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)UserWorld 'python' 2733760 with cmdline 'unknown', parent 2733744
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)started from 'python' 2733744 with cmdline '/usr/bin/python /usr/lib/vmware/nsx-esx-datapath/bin/jumpstart.py', parent 2733743
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)started from 'sh' 2733743 with cmdline '/bin/sh -c LANG=en_US.UTF-8  /usr/lib/vmware/nsx-esx-datapath/bin/jumpstart.py
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel:    ', parent 2733742
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)started from 'jumper2' 2733742 with cmdline '/sbin/jumper2 --plugin=nsx-datapath.json', parent 2733741
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)started from 'jumper2' 2733741 with cmdline '/sbin/jumper2 --plugin=nsx-datapath.json', parent 2733219
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)started from 'python' 2733219 with cmdline '/usr/bin/python /usr/lib/vmware/lifecycle/bin/imagemanagerctl.py software --apply --task-id 526c9929-83b3-2ffa-4bc9-200bc22916d4 --depot http://hostname.$
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)started from 'init' 2097869 with cmdline '/bin/init', parent 0
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: cpu17:2733744)uw.2733760 (3341962) requires 700 KB, asked 700 KB from python.2733219 (3337353) which has 322220 KB occupied and 340 KB available.

       

  • SDDC Manager UI will report following error :


  • ESXi host upgrade in SDDC Manager will also fail with the following SHA1 certificate error even though there is no SHA1 certificate present on ESXI host and the vCenter.

    "SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate. Refer to the release notes and KB 89424 for more details."




Environment

  • VMware vSphere 8.0 U3
  • VMware Cloud Foundation 5.2.x

Cause

This issue occurs due to an insufficiently sized memory pool for an ESXi service leads to VIB installation failures, ultimately causing the NSX deployment or ESXi upgrade to fail.

Resolution

This is a known issue and it is resolved in ESXi 8.0 update 3g build 24859861. 

Reference: VMware ESXi 8.0 Update 3g Release Notes

Workaround:

  • Login to the ESXi host via SSH
  • Run the command below on the affected host(s) to increase the memory to 400 MB.
    • localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb
    • Note: In some scenario you might need to increase the memory more than 400 MB, accordingly you will increase the memory for the pool.

  • Perform the remediation of the image to install the NSX VIBs on the ESXi host.

NOTE: Please reboot the ESX host after any failed remediation attempt. Remediation will still fail even after performing the above workaround if the host was not rebooted since last failed attempt. 

Powered by Wolken Software