"The host returns esxupdate error code:15" error when remediating an ESXi 6.x and 7.x host

Products

• VMware vSphere ESXi

Issue/Introduction

• vSphere Lifecycle Manager (vLCM) fails to remediate ESXi 6.x or 7.x hosts in vSphere 7.0 and later
• This error may occur after updating VMware_locker_tools-light package.
• Error:
  error code:15. The package manager transaction is not successful. Check the Update Manager log files and esxupdate log files for more details.
  
  
• In the /var/log/esxupdate.log file, you see entries similar to:
  <YYYY-MM-DD>T<time> esxupdate: HostImage: DEBUG: --- Stage: LockerInstaller adding [VMware_locker_tools-light_5.0.0-1.12.653509], removing []</time>
<YYYY-MM-DD>T<time></time> esxupdate: HostImage: INFO: Attempting to download VIB tools-light
<YYYY-MM-DD>T<time></time><YYYY-MM-DD>T<time> esxupdate: downloader: DEBUG: Downloading from http://IP_ADDRESS:9084/vum/repository/hostupdate/vmw/vib20/tools-light/VMware_locker_tools-light_5.0.0-1.12.653509.vib...</time>
<YYYY-MM-DD>T<time></time> esxupdate: LockerInstaller: WARNING: There was an error in cleaning up product locker: [Errno 9] Bad file descriptor: '/locker/packages/5.0.0/floppies'
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: Traceback (most recent call last):</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: cmd.Run()</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 144, in Run</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 243, in InstallVibsFromSources</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 345, in _installVibs</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/Transaction.py", line 388, in _validateAndInstallProfile</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 630, in Stage</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-474610/bora/build/esx/release/python-2.6-lib-zip-stage/474610/visor/pylib/python2.6/site-packages/vmware/esximage/HostImage.py", line 463, in _download_and_stage
<YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: InstallationError: ('VMware_locker_tools-light_5.0.0-1.12.653509', '[Errno 32] Broken pipe')
<YYYY-MM-DD>T<time> esxupdate: esxupdate: DEBUG: <<<</time></time></time>
<YYYY-MM-DD>T<time> esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetTimeout']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.</time>
<YYYY-MM-DD>T<time> esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRetries']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.</time>
<YYYY-MM-DD>T<time> esxupdate: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-q', '-g', '/UserVars/EsximageNetRateLimit']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.</time>
<YYYY-MM-DD>T<time> esxupdate: esxupdate: INFO: ---</time>
  
  or
  
  <YYYY-MM-DD>T<time> esxupdate: esxupdate: ERROR: DatabaseIOError: ('/locker/packages/var/db/locker', "Failed to create temporary DB dir: [Errno 28] No space left on device: '/locker/packages/var/db/locker/vibs.new'"
   
• The issue persists even after the host is rebooted.

Environment

vSphere Exsi version 7.0

vSphere Exsi version 8.0

Cause

This issue occurs if the /locker/ folder contains too many files aside from the /locker/packages (or /store/packages) folder. Remediation cannot happen as the disk does not have enough space.

Resolution

To resolve this error when remediating the upgrade baseline, cleanup /locker/ folder.

1. Use df /locker to find how much the disk has been consumed. After the failed remediation, no tools-light VIB would remain on the disk. If at this point the disk is still more than half used, a cleanup is necessary.
2. Run du -ah /locker/* to find out what are in the locker, remove large files in the directory.
3. One potential usage of /locker is vSAN traces, to setup trace in a datastore or other storage see The ramdisk vsantraces" is full" error reports in vSAN logging
4. After cleaning up the partition, retry the scan/remediation.