Unable to patch or upgrade an ESXi host as bootbank is pointed to /tmp
search cancel

Unable to patch or upgrade an ESXi host as bootbank is pointed to /tmp

book

Article ID: 426834

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

When attempting to patch or upgrade an ESXi host using VMware vSphere Lifecycle Manager (vLCM), the process fails. You will observe the following symptoms:

  • vLCM/vCenter Error:

    "An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 30."

    Could not stage image profile '(Updated) DEL-ESXi_803.<build_number>':The transaction is not supported: VIB VMware_bootbank_gc_8.0.3-0.88.<build_number> cannot be live installed. VIB VMware_bootbank_esx-update_8.0.3-<build_number> cannot be live installed. VIB VMware_bootbank_esx-base_8.0.3-<build_number>cannot be live installed. VIB VMware_bootbank_vsan_8.0.3-<build_number>cannot be live installed. VIB VMware_bootbank_native-misc-drivers_8.0.3-<build_number>cannot be live installed. VIB VMW_bootbank_vmkusb_0.1-24vmw.<build_number>cannot be live installed. VIB VMware_bootbank_gc_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_esx-update_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_esx-dvfilter-generic-fastpath8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_esx-base_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_native-misc-drivers_8.0.3-<build_number>cannot be removed live. VIB VMware_bootbank_vsan_8.0.3-<build_number>cannot be removed live. VIB VMW_bootbank_vmkusb_0.1-23vmw.<build_number>cannot beremoved live.

  • Error messages indicating that VIBs (e.g., esx-base, vsan, vmkusb) cannot be live installed or removed.

  • Running ls -l / shows the bootbank symlink pointing to a temporary directory in /tmp instead of a physical partition: bootbank -> /tmp/bootbank#######

  • The SD-Card or USB boot media is visible in the physical server console (iDRAC/iLO) and via ISO boot, but is missing from the ESXi command line (esxcli storage core device list).

    • hostd.log (/var/log/hostd.log) records the failure of the InstallV2 task with exit code 30:
      YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099334]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 1352 : Could not stage image profile ' (Updated) DEL-ESX1 803.
24022510-A00': The transaction is not supported: VIB VMware_bootbank_esx-update 8.0.3-0.88.<build_number> cannot be live installed. VIB VMW_bootbank_vmkusb_0.1-24v
mw.803.0.88.<build_number> cannot be live installed. VIB VMware_bootbank_gc 8.0.3-0.88.<build_number> cannot be live installed. VIB VMware_bootbank_native-misc-drivers_8
be live installed. VIB VMware_bootbank_esx-base 8.0.3-0.88.2
5067014 cannot be live installed. VIB VMware bootbank_esx-dvfilter-generic-fastpath 8.0.3-0.79.24859861 cannot be removed live. VIB VMware_bootbank_esx-base
be removed live. VIB VMware_bootbank_vsan 8.0.3-0.79.24859861
be removed live. VIB VMW_bootbank vmkusb 0.1-23vmw.803.0.70.2467446439
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099338]: [Originator@6876 sub=Vimsvc. TaskManager opID-3f290071-da-24b5 sid-52220537 user-vpxuser:com. vmware. vcIntegri
ty] Task Completed : haTask-ha-host-vim.host. PatchManager. InstallV2-1886422344 Status error
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099338]: [Originator@6876 sub-Hostsvc. PatchManager opID-3f290071-da-24b5 sid-52220537 user=vpxuser:com.vmware.vcInteg
rity] PatchManager InstallV2 task returned: 30
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099336]: [Originator@6876 sub-Vimsvc.TaskManager opID-57986eal-87-24fl sid-52220537 user=vpxuser:com. vmware.vcIntegri
ty] Task Created : haTask-ha-host-vim.host. PatchManager. ScanV2-1886422367
YYYY-MM-DDTHH:MM:SS  In(166) Hostd[2099335]: [Originator@6876 sub-SysCommandPosix opID-57986eal-87-24fl sid=52220537 user=vpxuser:com.vmware.vcIntegrity]
ForkExec (/usr/sbin/esxupdate) 2115256
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099323]: [Originator@6876 sub=Hostsvc. VmkVprobSource] VmkVprobSource :: Post event: (vim.event. EventEx) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    key = 183,
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    chainId = -1702579664, STATTC39
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    createdTime = "1970-01-01T00:00:00Z",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    userName = <user_name>
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    host - (vim.event. HostEventArgument) {
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    name = "<esxi_host_fqdn>",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    host = 'vim. HostSystem:ha-host'
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    eventTypeId = "esx.audit.net.firewall.config.changed",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    arguments = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    (vmodl. KeyAnyValue) (
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    key = "1",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    value - "enable"
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    (vmodl.KeyAnyValue) (
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    key = "2",
YYYY-MM-DDTHH:MM:SS In(166) Hostd[2099314]: -- >    value = "esxupdate"

       

    • /var/log/esxupdate.log

      The underlying cause is identified here—the installer cannot find the primary or secondary bootbank

      YYYY-MM-DDTHH:MM:SS In(14) esxupdate[2115256]: Installer <class 'esximage. Installer. BootBankInstaller.BootBankInstaller'> was not initiated - reason: altbootbank is invalid: Error in loading boot.cfg from bootbank /bootbank: [Errno 2] No such file or directory: '/bootbank/boot.cfg'
YYYY-MM-DDTHH:MM:SS In(14) esxupdate[2115256]: Installers initiated are {'quickpatch': <esximage. Installer. QuickPatchInstaller. QuickPatchInstaller object at 0xb3d23970d0>, 'live': <esximage. Installer. LiveImageInstaller. LiveImageInstaller object at 0xb3dl1f61d0>}
YYYY-MM-DDTHH:MM:SS In(14) esxupdate [2115256]: runcommand called with: args = ['/sbin/smbiosDump'], outfile = None, returnoutput = True, timeout = 0.0.
YYYY-MM-DDTHH:MM:SS In(14) esxupdate [2115256]: Adding VIB VMware_bootbank_trx_8.0.3-0.88.<build_number> to ImageProfile (Updated) DEL-ESXi_803.<build_number>-A00
YYYY-MM-DDTHH:MM:SS Db(15) esxupdate [2115256]: VIBs ('VMware_bootbank_trx 8.0.3-0.79.<build_number>') are replaced by VIB VMware_bootbank_trx_8.0.3-0.88.<build_number>, removing them from ImageProfile (Updated) DEL-ESX1 803.<build_number>-A00
  • Inbuilt SDCard of 64 GB storage used in the ESXi is not seen by the host.
  • boot.log and jumpstart logs have been scanned and there is no reference of vmhba33 (SD-Card adapter), however the same is visible from iDRAC/iLO and also during upgrade process using ISO it is visible, but not seen within ESXi host.

Environment

VMware vSphere ESXi 7.x 
VMware vSphere ESXi 8.x

Cause

This issue occurs when the vmkusb module is disabled on the ESXi host.

In modern ESXi versions, many internal SD-card readers and USB boot devices rely on the vmkusb driver to be detected. If this module is disabled, the hypervisor cannot claim the boot device during the "jumpstart" process. Consequently, ESXi mounts a temporary bootbank in RAM (/tmp), which lacks the persistence required for patching or upgrades.

Resolution

To resolve this issue, you must re-enable the vmkusb module, force the ESXi device manager to re-scan for the boot device, and perform a domain rejoin to refresh stale credentials.

  1. Enter Maintenance Mode Place the ESXi host into Maintenance Mode via the vSphere Client or the command line.

  2. Enable the Module and Restart Device Manager Run the following commands to enable the vmkusb module and refresh the device manager process:
    esxcli system module set -m vmkusb -e truekill -SIGHUP $(ps -C | grep vmkdevmgr | awk '{print $1}')

  3. Verify Device Visibility Confirm if the USB-attached boot devices are now visible to the system:
    lsusb# OResxcli storage core device list
  4. Confirm Module Status Verify that the module is correctly loaded and enabled:
    esxcli system module list | grep vmkusb
    Expected Output: vmkusb true true

  5. Persist Configuration to Physical Storage Since the bootbank was redirected to /tmp, you must manually save the "Enabled" state to physical storage once the device is detected. Identify the active bootbank and back up the state:
    cp /bootbank/state.tgz /vmfs/volumes/<ACTIVE_BOOTBANK_VOLUME>/

  6. Refresh Active Directory Credentials (vSphere UI) To resolve stale credential issues (such as LW_ERROR_PASSWORD_MISMATCH) resulting from the bootbank redirection, use the following steps in the vSphere Client:

    • Leave the Domain:

      1. In the vSphere Client, navigate to the ESXi host.
      2. Click the Configure tab.
      3. Under System, select Authentication Services.
      4. Click Leave Domain.
  7. Reboot to ensure the vmkusb driver loads at startup and the bootbank points to the correct physical partition.

  8.  Rejoin the Domain:

      1. Once the host is back online, return to Configure > System > Authentication Services.
      2. Click Join Domain.
      3. Enter the Domain name and User credentials (using the format user@domain.com or domain\user).
      4. Click OK to synchronize the machine account password.
Powered by Wolken Software