ESXi patching pre-check fails with SHA-1 signature error due to memory exhaustion
search cancel

ESXi patching pre-check fails with SHA-1 signature error due to memory exhaustion

book

Article ID: 433124

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • During the ESXi 8.0 patching process, the pre-check fails and displays the following error string: SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate. Refer to release notes and KB 89424 for more details.

  • vCenter Server and the connected ESXi hosts are not using certificates with a weak digital signature algorithm. Running the vsphere8_upgrade_certificate_checks.py script (from KB 313460) verifies this configuration and detects no certificates with a SHA-1 signature. Additionally, renewing and refreshing the ESXi certificates do not resolve the error.

  • The following entries are recorded in the vCenter Server /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log file:

    -->        "STRUCTURE": {
    -->            "com.vmware.esx.settings.notification": {
    -->                "id": "com.vmware.vcIntegrity.lifecycle.HostScan.UnsupportedSHA1Cert",
    -->                "message": {
    -->                    "STRUCTURE": {
    -->                        "com.vmware.vapi.std.localizable_message": {
    -->                            "args": [
    -->                                "False"
    -->                            ],
    -->                            "default_message": "SHA-1 signature found in host certificate False. Support for certificates with weak signature algorithm SHA-1 has been removed in ESXi 8.0. To proceed with upgrade, replace it with a SHA-2 signature based certificate. Refer to the release notes and KB 89424 for more details.",
    -->                            "id": "com.vmware.vcIntegrity.lifecycle.HostScan.UnsupportedSHA1Cert",
    -->                            "localized": {
    -->                                "OPTIONAL": null

  • The following log excerpts appear in the ESXi host logs:
    • Note: The memory exhaustion message only appears immediately after a pre-check is triggered, so it may not be visible in recent logging depending on when the last check was performed.

    • /var/run/log/lifecycle.log:

      YYYY-MM-DDTHH:MM:SS Er(11) lifecycle[6493677]: upgrade_precheck:2924 Failed to parse certificate /etc/vmware/ssl/castore.pem: Command openssl x509 -in /tmp/tmp9c11f5v4 -noout -text | grep 'Signature Algorithm' exited with code 2
      YYYY-MM-DDTHH:MM:SS Er(11) lifecycle[2217260]: upgrade_precheck:2924 Failed to parse certificate /etc/vmware/ssl/castore.pem: openssl x509 -in /tmp/tmpi99ir2fb -noout -text | grep 'Signature Algorithm' failed to execute: Error running command "openssl x509 -in /tmp/tmpi99ir2fb -noout -text | grep 'Signature Algorithm'": [Errno 12] Cannot allocate memory

      or

      YYYY-MM-DDTHH:MM:SS In(14) lifecycle[14502278]: upgrade_precheck:1521 Running command openssl x509 -in /tmp/tmpnegrm5kw -noout -text | grep 'Signature Algorithm'
      YYYY-MM-DDTHH:MM:SS In(14) lifecycle[14502278]: runcommand:199 runcommand called with: args = "openssl x509 -in /tmp/tmpnegrm5kw -noout -text | grep 'Signature Algorithm'", outfile = None, returnoutput = True, timeout = 0.0.
      YYYY-MM-DDTHH:MM:SS Er(11) lifecycle[14502278]: upgrade_precheck:2924 Failed to parse certificate /etc/vmware/ssl/castore.pem: Command openssl x509 -in /tmp/tmpnegrm5kw -noout -text | grep 'Signature Algorithm' exited with code 2

    • /var/run/log/vmkernel.log:

      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.14502278:python.14502623:uw.14502623
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)UserWorld 'python' 14502623 with cmdline 'unknown', parent 14502278
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'python' 14502278 with cmdline '/usr/bin/python /usr/lib/vmware/lifecycle/bin/imagemanagerctl.py software --scan --software-spec /var/run/lifecycle.scan.swspec --depot http://VCENTER.FQDN$
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'init' 2098019 with cmdline '/bin/init', parent 0
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)uw.14502623 (97277664) requires 752 KB, asked 752 KB from python.14502278 (97274898) which has 306452 KB occupied and 748 KB available.
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.14502278:python.14502623:uw.14502623
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)UserWorld 'python' 14502623 with cmdline 'unknown', parent 14502278
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'python' 14502278 with cmdline '/usr/bin/python /usr/lib/vmware/lifecycle/bin/imagemanagerctl.py software --scan --software-spec /var/run/lifecycle.scan.swspec --depot http://VCENTER.FQDN$
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)started from 'init' 2098019 with cmdline '/bin/init', parent 0
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu13:14502278)uw.14502623 (97277664) requires 752 KB, asked 752 KB from python.14502278 (97274898) which has 306452 KB occupied and 748 KB available.
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu13:14502278)WARNING: LinuxThread: 421: python: Error cloning thread: -28 (bad0081)
      ...
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 3892: Could not allocate 12288 bytes for dynamic heap worldGroup.14503226. Request returned Admission check failed for memory resource
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 4105: Heap_Align(worldGroup.14503226, 10408/10408 bytes, 64 align) failed.  caller: 0x42002354e1db
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: LinuxThread: 421: sh: Error cloning thread: -12 (bad0014)
      ...
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu115:14503224)worldGroup.14503226 (97282278) requires 4 KB, asked 4 KB from python.14502278 (97274898) which has 307200 KB occupied and 0 KB available.
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu115:14503224)Admission failure in path: host/vim/vmvisor/settingsd-task-forks/python.14502278:sh.14503226:worldGroup.14503226
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu115:14503224)worldGroup.14503226 (97282278) requires 4 KB, asked 4 KB from python.14502278 (97274898) which has 307200 KB occupied and 0 KB available.
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 3892: Could not allocate 12288 bytes for dynamic heap worldGroup.14503226. Request returned Admission check failed for memory resource
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: Heap: 4105: Heap_Align(worldGroup.14503226, 10408/10408 bytes, 64 align) failed.  caller: 0x42002354e1db
      YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu115:14503224)WARNING: LinuxThread: 421: sh: Error cloning thread: -12 (bad0014)
      YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu113:14503233)FSS: 8869: Unmounting file descriptor 430cb89da4f0

       

Environment

  • ESXi 8.x

Cause

Memory exhaustion in the settingsd-task-forks resource pool causes the task to fail.

This exhaustion prevents the host from executing the OpenSSL certificate validation command during the pre-check, resulting in a false positive SHA-1 certificate error.

Resolution

This issue is fixed in vSphere ESXi 8.0 Update 3g and later releases.

Reference: VMware ESXi 8.0 Update 3g Release Notes

Note : While ESXi 8.0 U3g increases the default memory allocation for the settingsd pool to 345 MB, environments with heavier configurations running 8.0 U3g or later (such as 8.0 U3i) may still exceed this new limit and experience the false positive SHA-1 error. If observed on 8.0 U3g or later, proceed with the Workaround (Option 1) to manually increase the memory limit to 400 MB.

Workaround:

Option 1 - Increase the memory limit of the host to allow the update to a version with the fix:

  1. Log in to the affected ESXi host as root via SSH or the ESXi Shell.

  2. Check the current memory configuration for the settingsd-task-forks group on the affected ESXi host by executing the following command:
    localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group getmemconfig -g host/vim/vmvisor/settingsd-task-forks
       
  3. Increase the memory limit to 400 MB by  executing the following command:
    localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/settingsd-task-forks -m 400 -i 0 -l -1 -u mb
        
  4. Confirm the updated memory configuration by re-executing the validation command from Step 2.

  5. Re-initiate the patching operation.

Note : Incase the issue persists after increasing to 400 then increase the memory limit to 500

Option 2 - Force the update the host to a version with the fix using the profile update command line alternative:

  1. Enter the host to maintenance mode.
  2. Reboot the host.
  3. Use the steps of KB 343840 (Patching ESXi host using Command Line) to force the update of the host (parameter --force in the command)
  4. Reboot the host again.
  5. Validate the host is now updated.
  6. Exit Maintenance Mode

Note: If there is SDDC Manager managing the vSphere, the KB 390727 (SDDC Manager Precheck: Out-Of-Band Upgrade Error)  must be considered after forcing the update using command line.

Additional Information

Powered by Wolken Software