ESXi remediation failing due to settingsd-task-forks errors

Products

• VMware vSphere ESXi

Issue/Introduction

When performing host remediation or image updates via vSphere Lifecycle Manager (vLCM), the task fails during the execution of Python scripts. This often occurs during the stagebootbank mounting phase or when applying specific VIB components.

 

The Remediation task fails:

• The host lifecycle.log will report error's parsing the vibs.
  

  YYYY-MM-DDTHH:MM:SSZ Db(15) lifecycle[###]: HostImage:#### installer BootBankInstaller failed: secureMount returns status 255, output: Ignoring error when loading bootbank: Error in loading database for bootbank /usr/lib/vmware/lifecycle/stagebootbank: ('/usr/lib/vmware/lifecycle/stagebootbank/imgdb.tgz', "Error parsing VIB/ImageProfile from DB /usr/lib/vmware/lifecycle/stagebootbank/imgdb.tgz: VibFormatError - ('Unknown', 'Could not parse VIB XML data: unknown error, line 1, column 145826 (<string>, line 1).')")

   

• In the /var/run/log/vmkernel.log on the impacted ESXi host, entries indicate memory allocation failures for python or lifecycle processes: 
  

  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: ... requires <X> KB, asked <X> KB from python.<PID> which has <Y> KB occupied and <Z> KB available.

 

Cause

This is caused by  a know memory limit in the settingsd-task-forks service.

Resolution

This issue is resolved in vSphere ESXi 8.0 Update 3g and later releases.

Reference: VMware ESXi 8.0 Update 3g Release Notes

The memory allocation for the settingsd-task-forks service has been increased to 345 MB to accommodate Python-based remediation tasks.

Workaround:

• Run the below command to increase the memory to 400 MB.

  • localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb

  • Note: In some scenario there might be a need to increase the memory more than 400 MB, accordingly it will increase the memory for the pool.
• Perform the remediation of the image again.

Additional Information

For NSX remediation tasks failing with this error see KB: https://knowledge.broadcom.com/external/article/434545/nsx-upgrade-fails-during-vlcm-remediatio.html