Error: "Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk" while remediating the ESX host due to settingsd-task-forks admission failure

Products

VMware vSphere ESXi

Issue/Introduction

When attempting to perform lifecycle management operations (Check Compliance, Stage, or Remediate) on an ESXi host using vSphere Lifecycle Manager (vLCM), the following symptoms occur:

While performing Host compliance check: It is reported error : Host status is unknown and " An internal error occurred while staging/remediating host."

In some cases, Pre-check fails with false-positive SHA-1 certificate error.

"SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate."
/var/run/log/lifecycle.log on the ESXi host may have the following errors are recorded:

YYYY-MM-DDTHH:MM:SS Wa(12) lifecycle[####]: BootBankInstaller:249 Ignoring error when loading bootbank: Error in loading database for bootbank /altbootbank: ('/altbootbank/imgdb.tgz', 'Error parsing VIB/ImageProfile from DB /altbootbank/imgdb.tgz: MemoryError - ')

YYYY-MM-DDTHH:MM:SSZ Er(11) lifecycle[8965070]: QuickPatchInstaller:1457 Unexpected error occurred while attempting to run Quick Patch scan scripts: Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk.
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: File "/lib64/python3.11/site-packages/vmware/esximage/Installer/QuickPatchInstaller.py", line 1436, in _runScanScripts
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: Ramdisk.MountTardiskInRamdisk(downloadPath,
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: File "/lib64/python3.11/site-packages/vmware/esximage/Utils/Ramdisk.py", line 229, in MountTardiskInRamdisk
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: raise Errors.InstallationError(None, None, msg)
YYYY-MM-DDTHH:MM:SSZ Er(11)[+] lifecycle[8965070]: vmware.esximage.Errors.InstallationError:Current ESXi version does not provide a mechanism to mount a tardisk into a ramdisk.
In /var/run/log/vmkernel.log on ESXi host, admission failure errors are observed:

YYYY-MM-DDThh.mm.ss In(###) vmkernel: cpu#.####)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.####:sh.####:worldGroup.####

Environment

VMware vSphere ESXI 8.0 U3h and higher versions.
vCenter Server 8.0 U3.

Cause

The ESXi host encounters memory exhaustion within the host/vim/vmvisor/settingsd-task-forks resource group. The default memory limit is insufficient for the vLCM daemon to parse large VIB/ImageProfile databases (imgdb.tgz) during the upgrade/remediation process.

Resolution

This is a known issue with VMware vSphere ESXi 8.0. It is targeted to be fixed in a future release. To be notified of updates, subscribe to this article.

NOTE: If the compliance has already failed with an error message like the following, please follow https://knowledge.broadcom.com/external/article/451408/image-compliance-scan-fails-during-the-v.html before executing this KB.

com.vmware.vapi.std.errors.not_allowed_in_current_state Messages: com.vmware.esx.task.contention<Another task is in progress. Please retry later.>

Workaround

Perform any of the below methods to work around this issue:

Method 1 - Execute depot override script on vCenter Server.

Login to vSphere Client and select the vSphere Cluster.
Note down the MoID of the Cluster, for eg. "domain-c9" in below sample screenshot:
Download the "depot-override.sh" script attached to this KB.
Copy the script to vCenter Server using WinSCP or copy/paste the contents by creating new file on vCenter using 'vi' editor.
Grant execution permission.

[ ~ ]# vi depot-override.sh
[ ~ ]# chmod +x depot-override.sh
Execute the script to configure depot override.

./depot-override.sh configure --clusters <Cluster MoID>
1. Important Notes:
  1. Remove the depot override configuration after successful patching operation, details in Step 9.
  2. The depot override will ONLY work for the EXACT desired state that it was exported from. If the desired state is changed (eg. added a new addon/component or enabled NSX/HA), then the depot override needs to be configured again. Remove the existing depot override and configure a new one if the desired state has changed.
  3. If the depot override is not sufficient for a desired state, operations will fail with error "Vendor addon <AddonName> with version <ESX version > cannot be found in depot."

Enter SSO Administrator credentials and vCenter FQDN when prompted by the script.

[ ~ ]# ./depot-override.sh configure --clusters domain-c9
vCenter username [[email protected]]: [email protected]
vCenter password:
Logging to /tmp/depot-override-######-######.log
vCenter FQDN [vcenter.example.com]: vcenter.example.com

Sample execution result:

[ ~ ]# ./depot-override.sh configure --clusters domain-c9
vCenter username [[email protected]]: [email protected]
vCenter password:
Logging to /tmp/depot-override-######-#######.log
vCenter FQDN [vcenter.example.com]: vcenter.example.com

==== Exporting desired-image offline bundle for cluster domain-c9 ====
$ dcli com vmware esx settings clusters software export --cluster domain-c9 --export-offline-bundle true --export-iso-image false --export-software-spec false
OFFLINE_BUNDLE: https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_#######-####-####-####-##############.zip
Bundle URL: https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_#######-####-####-####-##############.zip

==== Preparing depot folder /storage/updatemgr/patch-store/domain-c9-custom-depot and downloading bundle ====
$ curl -sk -o /storage/updatemgr/patch-store/domain-c9-custom-depot/offlineBundle.zip https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_#######-####-####-####-##############.zip

==== Unzipping offline bundle ====
$ unzip -o -q /storage/updatemgr/patch-store/domain-c9-custom-depot/offlineBundle.zip -d /storage/updatemgr/patch-store/domain-c9-custom-depot

==== Validating depot is served over HTTP ====
$ curl -sk -o /dev/null -w %{http_code} http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml
200
Depot reachable at http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml

==== Adding depot override for cluster domain-c9 ====
$ dcli com vmware esx settings clusters depotoverrides add --cluster domain-c9 --location http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml
.
.
.
==== Scan completed successfully for cluster domain-c9 ====
Depot override URL added for cluster domain-c9: http://vcenter.example.com:9084/vum/repository/domain-c9-custom-depot/index.xml

Re-initiate the patching operation.
Remove the depot override configuration after the patching operation by executing the script:

depot-override.sh remove --clusters <Cluster MoID>

Example: depot-override.sh remove --clusters domain-c9

Method 2 - Manually increase the memory allocation for the settingsd-task-forks resource pool.

Note: This action needs to be performed on each ESXi host.

Open an SSH session to the affected ESXi host as root user.
Check the current memory configuration for the settingsd-task-forks group on the affected ESXi host by executing the following command:

python -c ""from configRP import ResourcePool; rp = ResourcePool('host/vim/vmvisor/settingsd-task-forks'); print(rp.getCurrentMaxMemSize())""
Execute the following command to Increase the resource pool size such that the total now is 400MB. For eg: If the Step 2 returns 345MB, the increase should be 55MB. If it gives 315MB, the increase should be 85MB etc.:

python -c ""from configRP import ResourcePool; rp = ResourcePool('host/vim/vmvisor/settingsd-task-forks'); rp.increaseResourcePoolSize(0, 55)""Note: This command applies a temporary configuration that will not survive a reboot. This is the intended behavior, as the value should revert to its original state when the host reboots during patching.
Re-run the validation command to verify that the updated memory limit has been applied successfully.

python -c ""from configRP import ResourcePool; rp = ResourcePool('host/vim/vmvisor/settingsd-task-forks'); print(rp.getCurrentMaxMemSize())""
In the vCenter Server inventory, select the affected ESXi host, navigate to Updates, and click Check Compliance to refresh the host status.
Verify that the Actions menu is now fully accessible.
Re-initiate the patching operation.

Additional Information

Revert the resource pool configuration.

The resource pool memory increase must be temporary and apply only to the current patching attempt. Using localcli command to increase the resource pool is not recommended, as this makes the change persistent across reboots and will cause other issues.

Reset the value to default if "localcli" command was used on the hosts to increase the memory. The way to do it is outlined below:

SSH to the ESX host
Check the pool size of settingsd-task-forks.
1. cat /etc/rp.d/settingsd.yml and note down the resource pool size under host/vim/vmvisor/settingsd-task-forks
Execute below localcli command to set it to the value from step 2.a.

localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m value_from_step_2a -i 0 -l -1 -u mb

SHA256 Hash

depot-override.sh - 677DEA843DA9471B7F08BA556F35E7FC6050B49AD8082F9FA69B2580705CD78C

Attachments

depot-override.sh get_app