Error: "Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk" while remediating the ESX host due to settingsd-task-forks admission failure

Products

VMware vSphere ESXi

Issue/Introduction

When attempting to perform lifecycle management operations (Check Compliance, Stage, or Remediate) on an ESXi host using vSphere Lifecycle Manager (vLCM), the following symptoms occur:

The task fails with the error: "An internal error occurred while staging/remediating the host."
Pre-check fails with false-positive SHA-1 certificate error.

"SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate."
In some cases, /var/run/log/lifecycle.log on the ESXi host may have the following errors are recorded:

YYYY-MM-DDTHH:MM:SS Wa(12) lifecycle[####]: BootBankInstaller:249 Ignoring error when loading bootbank: Error in loading database for bootbank /altbootbank: ('/altbootbank/imgdb.tgz', 'Error parsing VIB/ImageProfile from DB /altbootbank/imgdb.tgz: MemoryError - ')

YYYY-MM-DDTHH:MM:SSZ Er(11) lifecycle[8965070]: QuickPatchInstaller:1457 Unexpected error occurred while attempting to run Quick Patch scan scripts: Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk.
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: File "/lib64/python3.11/site-packages/vmware/esximage/Installer/QuickPatchInstaller.py", line 1436, in _runScanScripts
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: Ramdisk.MountTardiskInRamdisk(downloadPath,
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: File "/lib64/python3.11/site-packages/vmware/esximage/Utils/Ramdisk.py", line 229, in MountTardiskInRamdisk
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: raise Errors.InstallationError(None, None, msg)
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: vmware.esximage.Errors.InstallationError: Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk.
In /var/run/log/vmkernel.log on ESXi host, admission failure errors are observed:

YYYY-MM-DDThh.mm.ss In(###) vmkernel: cpu#.####)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.####:sh.####:worldGroup.####

Environment

vSphere ESXi 8.0 U3h and higher versions.
vCenter Server 8.0 U3.

Cause

The ESXi host encounters memory exhaustion within the host/vim/vmvisor/settingsd-task-forks resource group. The default memory limit is insufficient for the vLCM daemon to parse large VIB/ImageProfile databases (imgdb.tgz) during the upgrade/remediation process.

Resolution

This is a known issue with VMware vSphere ESXi 8.0. It is targeted to be fixed in a future release. To be notified of updates, subscribe to this article.

Workaround

Perform any of the below methods to work around this issue:

Method 1 - Execute depot override script on vCenter Server.

Login to vSphere Client and select the vSphere Cluster.
Note down the MoID of the Cluster, for eg. "domain-c9" in below sample screenshot:
Download the "depot-override.sh" script attached to this KB.
Copy the script to vCenter Server using WinSCP or copy/paste the contents by creating new file on vCenter using 'vi' editor.
Grant execution permission.

[ ~ ]# vi depot-override.sh
[ ~ ]# chmod +x depot-override.sh
Execute the script to configure depot override.

./depot-override.sh configure --clusters <Cluster MoID>
1. Important Notes:
  1. Remove the depot override configuration after successful patching operation, details in Step 9.
  2. The depot override will ONLY work for the EXACT desired state that it was exported from. If the desired state is changed (eg. added a new addon/component or enabled NSX/HA), then the depot override needs to be configured again. Remove the existing depot override and configure a new one if the desired state has changed.
  3. If the depot override is not sufficient for a desired state, operations will fail with error "Vendor addon <AddonName> with version <ESX version > cannot be found in depot."

Enter SSO Administrator credentials and vCenter FQDN when prompted by the script.

[ ~ ]# ./depot-override.sh configure --clusters domain-c9
vCenter username [[email protected]]: [email protected]
vCenter password:
Logging to /tmp/depot-override-######-######.log
vCenter FQDN [vcenter.example.com]: vcenter.example.com

Sample execution result:

[ ~ ]# ./depot-override.sh configure --clusters domain-c9
vCenter username [[email protected]]: [email protected]
vCenter password:
Logging to /tmp/depot-override-######-#######.log
vCenter FQDN [vcenter.example.com]: vcenter.example.com

==== Exporting desired-image offline bundle for cluster domain-c9 ====
$ dcli com vmware esx settings clusters software export --cluster domain-c9 --export-offline-bundle true --export-iso-image false --export-software-spec false
OFFLINE_BUNDLE: https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_#######-####-####-####-##############.zip
Bundle URL: https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_#######-####-####-####-##############.zip

==== Preparing depot folder /storage/updatemgr/patch-store/domain-c9-custom-depot and downloading bundle ====
$ curl -sk -o /storage/updatemgr/patch-store/domain-c9-custom-depot/offlineBundle.zip https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_#######-####-####-####-##############.zip

==== Unzipping offline bundle ====
$ unzip -o -q /storage/updatemgr/patch-store/domain-c9-custom-depot/offlineBundle.zip -d /storage/updatemgr/patch-store/domain-c9-custom-depot

==== Validating depot is served over HTTP ====
$ curl -sk -o /dev/null -w %{http_code} http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml
200
Depot reachable at http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml

==== Adding depot override for cluster domain-c9 ====
$ dcli com vmware esx settings clusters depotoverrides add --cluster domain-c9 --location http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml
.
.
.
==== Scan completed successfully for cluster domain-c9 ====
Depot override URL added for cluster domain-c9: http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml

Re-initiate the patching operation.
Remove the depot override configuration after the patching operation by executing the script:

depot-override.sh remove --clusters <Cluster MoID>

Example: depot-override.sh remove --clusters domain-c9

Method 2 - Manually increase the memory allocation for the settingsd-task-forks resource pool.

Note: This action needs to be performed on each ESXi host.

Open an SSH session to the affected ESXi host as root user.
Check the current memory configuration for the settingsd-task-forks group on the affected ESXi host by executing the following command:

localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group getmemconfig -g host/vim/vmvisor/settingsd-task-forks
Execute the following command to increase the memory limit of the settingsd-task-forks group to 400MB:

localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb

Note: If the issue persists, incrementally increase the resource limit (e.g., to 500 MB) to determine the minimum stable memory allocation required for the host environment.
Re-run the validation command from Step 2 to verify that the updated memory limit has been applied successfully.
In the vCenter Server inventory, select the affected ESXi host, navigate to Updates, and click Check Compliance to refresh the host status.
Verify that the Actions menu is now fully accessible.
Re-initiate the patching operation.

Additional Information

SHA256 Hash

depot-override.sh - 677DEA843DA9471B7F08BA556F35E7FC6050B49AD8082F9FA69B2580705CD78C

Attachments

depot-override.sh get_app