ESXi remediation failing due to settingsd-task-forks errors

Products

• VMware vSphere ESXi

Issue/Introduction

When performing host remediation or image updates via vSphere Lifecycle Manager (vLCM), the task fails during the execution of Python scripts. This often occurs during the stagebootbank mounting phase or when applying specific VIB components.

The Remediation task fails with the following errors:

• The host /var/log/lifecycle.log will report errors parsing the vibs.
  YYYY-MM-DDTHH:MM:SSZ Db(15) lifecycle[###]: HostImage:#### installer BootBankInstaller failed: secureMount returns status 255, output: Ignoring error when loading bootbank: Error in loading database for bootbank /usr/lib/vmware/lifecycle/stagebootbank: ('/usr/lib/vmware/lifecycle/stagebootbank/imgdb.tgz', "Error parsing VIB/ImageProfile from DB /usr/lib/vmware/lifecycle/stagebootbank/imgdb.tgz: VibFormatError - ('Unknown', 'Could not parse VIB XML data: unknown error, line 1, column 145826 (<string>, line 1).')")
• In the /var/run/log/vmkernel.log on the impacted ESXi host, entries indicate memory allocation failures for python or lifecycle processes: 
  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: ... requires <X> KB, asked <X> KB from python.<PID> which has <Y> KB occupied and <Z> KB available.

Environment

ESXi 8.x

Cause

A known memory limit exists in the settingsd-task-forks service. The default resource pool allocation is insufficient for certain large Python-based remediation tasks, particularly when parsing large imgdb.tgz files or extracting numerous VIBs.

Resolution

In vSphere ESXi 8.0 Update 3g, the default memory allocation for the settingsd-task-forks service was increased to 345 MB, however the issue has been observed again in ESXi 8.0 Update 3k

Reference: VMware ESXi 8.0 Update 3g Release Notes

Workaround:

1. Establish an SSH session to the affected ESXi host.
2. Increase the memory limit to 400 MB by running the following command:
   localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb
3. Note: In some scenarios, a limit higher than 400 MB may be required. Adjust the -m value accordingly if the MemoryError persists.
4. Re-initiate the host remediation task.

Additional Information

For NSX remediation tasks failing with this error see KB: https://knowledge.broadcom.com/external/article/434545/nsx-upgrade-fails-during-vlcm-remediatio.html