ESXi patching pre-check fails with SHA-1 signature error due to memory exhaustion

Products

• VMware vSphere ESXi

Issue/Introduction

• During the ESXi 8.0 patching process, the pre-check fails and displays the following error string: SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate. Refer to release notes and KB 89424 for more details.

• vCenter Server and the connected ESXi hosts are not using certificates with a weak digital signature algorithm. Running the vsphere8_upgrade_certificate_checks.py script (from KB 313460) verifies this configuration and detects no certificates with a SHA-1 signature. Additionally, renewing and refreshing the ESXi certificates do not resolve the error.

• The following entries are recorded in the vCenter Server /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log file:

  -->        "STRUCTURE": {
-->            "com.vmware.esx.settings.notification": {
-->                "id": "com.vmware.vcIntegrity.lifecycle.HostScan.UnsupportedSHA1Cert",
-->                "message": {
-->                    "STRUCTURE": {
-->                        "com.vmware.vapi.std.localizable_message": {
-->                            "args": [
-->                                "False"
-->                            ],
-->                            "default_message": "SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate. Refer to the release notes and KB 89424 for more details.",
-->                            "id": "com.vmware.vcIntegrity.lifecycle.HostScan.UnsupportedSHA1Cert",
-->                            "localized": {
-->                                "OPTIONAL": null

• The following log excerpts appear in the ESXi host logs:
  

  • /var/run/log/lifecycle.log:

    YYYY-MM-DDTHH:MM:SS Er(11) lifecycle[6493677]: upgrade_precheck:2924 Failed to parse certificate /etc/vmware/ssl/castore.pem: Command openssl x509 -in /tmp/tmp9c11f5v4 -noout -text | grep 'Signature Algorithm' exited with code 2
YYYY-MM-DDTHH:MM:SS Er(11) lifecycle[2217260]: upgrade_precheck:2924 Failed to parse certificate /etc/vmware/ssl/castore.pem: openssl x509 -in /tmp/tmpi99ir2fb -noout -text | grep 'Signature Algorithm' failed to execute: Error running command "openssl x509 -in /tmp/tmpi99ir2fb -noout -text | grep 'Signature Algorithm'": [Errno 12] Cannot allocate memory

    or

    YYYY-MM-DDTHH:MM:SS In(14) lifecycle[14502278]: upgrade_precheck:1521 Running command openssl x509 -in /tmp/tmpnegrm5kw -noout -text | grep 'Signature Algorithm'
YYYY-MM-DDTHH:MM:SS In(14) lifecycle[14502278]: runcommand:199 runcommand called with: args = "openssl x509 -in /tmp/tmpnegrm5kw -noout -text | grep 'Signature Algorithm'", outfile = None, returnoutput = True, timeout = 0.0.
YYYY-MM-DDTHH:MM:SS Er(11) lifecycle[14502278]: upgrade_precheck:2924 Failed to parse certificate /etc/vmware/ssl/castore.pem: Command openssl x509 -in /tmp/tmpnegrm5kw -noout -text | grep 'Signature Algorithm' exited with code 2

  • /var/run/log/vmkernel.log:
    
    YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.14502278:python.14502623:uw.14502623
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)UserWorld 'python' 14502623 with cmdline 'unknown', parent 14502278
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'python' 14502278 with cmdline '/usr/bin/python /usr/lib/vmware/lifecycle/bin/imagemanagerctl.py software --scan --software-spec /var/run/lifecycle.scan.swspec --depot http://VCENTER.FQDN$
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'init' 2098019 with cmdline '/bin/init', parent 0
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)uw.14502623 (97277664) requires 752 KB, asked 752 KB from python.14502278 (97274898) which has 306452 KB occupied and 748 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.14502278:python.14502623:uw.14502623
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)UserWorld 'python' 14502623 with cmdline 'unknown', parent 14502278
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'python' 14502278 with cmdline '/usr/bin/python /usr/lib/vmware/lifecycle/bin/imagemanagerctl.py software --scan --software-spec /var/run/lifecycle.scan.swspec --depot http://VCENTER.FQDN$
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'init' 2098019 with cmdline '/bin/init', parent 0
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)uw.14502623 (97277664) requires 752 KB, asked 752 KB from python.14502278 (97274898) which has 306452 KB occupied and 748 KB available.
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu13:14502278)WARNING: LinuxThread: 421: python: Error cloning thread: -28 (bad0081)
...
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 3892: Could not allocate 12288 bytes for dynamic heap worldGroup.14503226. Request returned Admission check failed for memory resource
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 4105: Heap_Align(worldGroup.14503226, 10408/10408 bytes, 64 align) failed.  caller: 0x42002354e1db
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: LinuxThread: 421: sh: Error cloning thread: -12 (bad0014)
...
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu115:14503224)worldGroup.14503226 (97282278) requires 4 KB, asked 4 KB from python.14502278 (97274898) which has 307200 KB occupied and 0 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu115:14503224)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.14502278:sh.14503226:worldGroup.14503226
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu115:14503224)worldGroup.14503226 (97282278) requires 4 KB, asked 4 KB from python.14502278 (97274898) which has 307200 KB occupied and 0 KB available.
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 3892: Could not allocate 12288 bytes for dynamic heap worldGroup.14503226. Request returned Admission check failed for memory resource
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 4105: Heap_Align(worldGroup.14503226, 10408/10408 bytes, 64 align) failed.  caller: 0x42002354e1db
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: LinuxThread: 421: sh: Error cloning thread: -12 (bad0014)
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu113:14503233)FSS: 8869: Unmounting file descriptor 430cb89da4f0

     

Environment

vSphere ESXi 8.0u3

Cause

The task fails due to memory exhaustion within the settingsd-task-forks resource pool on the ESXi host.

This exhaustion prevents the host from executing the OpenSSL certificate validation command during the pre-check, resulting in a false positive SHA-1 certificate error.

Resolution

This issue is resolved in vSphere ESXi 8.0 Update 3g and later releases.

Reference: VMware ESXi 8.0 Update 3g Release Notes

Workaround:

Option 1 - Increase the memory limit of the host to allow the update to a version with the fix:

1. Log in to the affected ESXi host as root via SSH or the ESXi Shell.
2. Check the current memory configuration for the settingsd-task-forks group on the affected ESXi host by executing the following command:
   localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group getmemconfig -g host/vim/vmvisor/settingsd-task-forks
3. Increase the memory limit to 400 MB by  executing the following command:
   localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb
4. Confirm the updated memory configuration by re-executing the validation command:
   localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group getmemconfig -g host/vim/vmvisor/settingsd-task-forks
5. Re-initiate the patching operation.

Option 2 - Force the update the host to a version with the fix using the profile update command line alternative:

1. Enter the host to maintenance mode.
2. Reboot the host.
3. Use the steps of KB 343840 (Patching ESXi host using Command Line) to force the update of the host (parameter --force in the command)
4. Reboot the host again.
5. Validate the host is now updated.
6. Exit Maintenance Mode

Note: If there is SDDC Manager managing the vSphere, the KB 390727 (SDDC Manager Precheck: Out-Of-Band Upgrade Error)  must be considered after forcing the update using command line.

Additional Information

• VMware ESXi 8.0 Update 3g Release Notes
• ESXi remediation failing due to settingsd-task-forks errors
• If there are no memory-related errors in the logging or the above steps do not resolve the issue, there might be a real problem with the certificates. See Upgrading vCenter Server or ESXi 8.0 fails during precheck due to a weak certificate signature algorithm for details on now to identify and resolve these issues.