ESXi host remediation or image updates via vSphere Lifecycle Manager (vLCM) fail with the error: "Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk" or the host status appears as "unknown."

Products

• VMware vSphere ESXi

Issue/Introduction

When performing host remediation or image updates via vSphere Lifecycle Manager (vLCM), the task fails during the execution of Python scripts. This often occurs during the stagebootbank mounting phase or when applying specific VIB components.

The Remediation task fails with the following errors:

Remediation failed:

Completed <date>, <time>

Remediation of cluster failed
Remediation failed for hosts <hostname>
<host-name> - Failed to remediate host
Remediation failed for host <hostname>
An internal error occurred while staging/remediating the host.

• The host /var/run/log/lifecycle.log will report errors parsing the vibs.
  YYYY-MM-DDTHH:MM:SSZ Db(15) lifecycle[###]: HostImage:#### installer BootBankInstaller failed: secureMount returns status 255, output: Ignoring error when loading bootbank: Error in loading database for bootbank /usr/lib/vmware/lifecycle/stagebootbank: ('/usr/lib/vmware/lifecycle/stagebootbank/imgdb.tgz', "Error parsing VIB/ImageProfile from DB /usr/lib/vmware/lifecycle/stagebootbank/imgdb.tgz: VibFormatError - ('Unknown', 'Could not parse VIB XML data: unknown error, line 1, column 145826 (<string>, line 1).')")
• In the /var/run/log/vmkernel.log on the impacted ESXi host, entries indicate memory allocation failures for python or lifecycle processes: 
  YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: ... requires <X> KB, asked <X> KB from python.<PID> which has <Y> KB occupied and <Z> KB available.

Environment

ESXi 8.x

Cause

A known memory limit exists in the settingsd-task-forks service. The default resource pool allocation is insufficient for certain large Python-based remediation tasks, particularly when parsing large imgdb.tgz files or extracting numerous VIBs.

Resolution

Broadcom is aware of the issue and a fix is expected in a future release.   

Workaround:

1. Establish an SSH session to the affected ESXi host.
2. Increase the memory limit to 600 MB by running the following command:
   
   localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 600 -i 0 -l -1 -u mb
   
   Note: In some scenarios, a limit higher than 600 MB may be required.  Adjust the -m value accordingly if the MemoryError persists.
   
   
3. Re-initiate the host remediation task.

Additional Information

For NSX remediation tasks failing with this error see KB: https://knowledge.broadcom.com/external/article/434545/nsx-upgrade-fails-during-vlcm-remediatio.html