NSX Upgrade Fails During VLCM Remediation Due to Insufficient Resource Pool Memory

Products

• VMware NSX

Issue/Introduction

When attempting to upgrade NSX on a vSphere Lifecycle Manager (vLCM) enabled cluster, the remediation process fails during the installation of VIBs. The following symptoms are observed:

• NSX deployment or upgrade status shows as failed in SDDC Manager or NSX Manager.

• The vLCM remediation task terminates unexpectedly with Install Failed
  
  

Remediation Failed

Remediation of cluster failed

Remediation failed for Host ###

Failed to remediate host [REDACTED]

Failed to apply Component nsx-lcp-bundle[REDACTED]: an error occurred while enabling service nsx-datapath

Remediation failed for Host ###

• SSH to impacted ESXi host root shell and navigate to /var/run/log/vmkernel.log  and check the below entries indicate memory allocation failures for python or lifecycle processes: 

  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: ... requires <X> KB, asked <X> KB from python.<PID> which has <Y> KB occupied and <Z> KB available.

Environment

VMware NSX

Cause

Insufficient sizing of the memory resource pool for settingsd-task-forks during the installation of VIBs in an NSX deployment prevents the lifecycle management processes from allocating necessary memory.

Resolution

To resolve this issue, increase the memory limit for the settingsd-task-forks resource pool on the affected ESXi host(s):

• Log in to the ESXi host CLI via SSH.
• Run the following command to increase the memory limit to 400 MB: localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb



Note: In some scenarios, it may be necessary to increase the memory further (e.g., to 500 MB or higher) depending on the specific resource demands of the VIB installation.

Retry the NSX upgrade or vLCM remediation task.

Additional Information

For ESXi host remediation tasks failing via vSphere Lifecycle Manager (vLCM) please refer to ESXi host remediation or image updates via vSphere Lifecycle Manager (vLCM) fail with the error: "Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk" or the host status appears as "unknown."