NSX Upgrade Fails During VLCM Remediation Due to Insufficient Resource Pool Memory

Products

• VMware NSX

Issue/Introduction

When attempting to upgrade NSX on a vSphere Lifecycle Manager (vLCM) enabled cluster, the remediation process fails during the installation of VIBs. The following symptoms are observed:

• NSX deployment or upgrade status shows as failed in SDDC Manager or NSX Manager.

• The vLCM remediation task terminates unexpectedly.
  

• SSH to impacted ESXi host root shell and navigate to /var/run/log/vmkernel.log  and check the below entries indicate memory allocation failures for python or lifecycle processes: 

  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: ... requires <X> KB, asked <X> KB from python.<PID> which has <Y> KB occupied and <Z> KB available.

Environment

• VMware NSX

Cause

• Insufficient sizing of the memory resource pool for settingsd-task-forks during the installation of VIBs in an NSX deployment prevents the lifecycle management processes from allocating necessary memory.

Resolution

To resolve this issue, increase the memory limit for the settingsd-task-forks resource pool on the affected ESXi host(s):

• Log in to the ESXi host CLI via SSH.
• Run the following command to increase the memory limit to 400 MB: localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb


• Note: In some scenarios, it may be necessary to increase the memory further (e.g., to 500 MB or higher) depending on the specific resource demands of the VIB installation.
  • Retry the NSX upgrade or vLCM remediation task.