Unable to Remediate Host due to Error while waiting for untar process

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

ESXi hosts booting from SD card (Note: SD Card revised guidance) may experience single image managed cluster remediation failures.

Remediation of cluster failed:

Remediation failed for Host : <esxi_hostname>

<esxi_hostname> - Failed to remediate host

In the lifecycle.log located at /var/run/log you see the following:

Er(11) lifecycle[3504558]: imagemanagerctl:101 esximage.Errors.InstallationError: VMware_locker_tools-light_12.3.5.22544099-23305545: Error while waiting for untar process '['/bin/tar', 'xzf', '-', '-C', '/locker/packages/']': Timeout (30 seconds) expired waiting for output from command '['/bin/tar', 'xzf', '-', '-C', '/locker/packages/']', pid '3504817'.

Cause

Due to performance issues on SD card level the untar process may fail due to time out.

Resolution

A combination of two workarounds may be needed to solve this issue:

Workaround 1:

Increase the timeout value of the untar process in the Lockerinstaller.py

1. Connect via SSH to ESXi
2. Run command:
   vsish -e set /config/VisorFS/intOpts/VisorFSPristineTardisk 0
3. Copy the LockerInstaller.py file to temporary location with following command, create a directory /backup in the /tmp.
   cp /lib64/python3.8/site-packages/vmware/esximage/Installer/LockerInstaller.py /tmp/backup/LockerInstaller.py
4. Now make a backup copy of the file with command:
   cp /tmp/backup/LockerInstaller.py /tmp/backup/LockerInstaller.py.bak
5. Correct permissions of the file:
   chmod 777 /tmp/backup/LockerInstaller.py
6. vi /tmp/backup/LockerInstaller.py so that below timeout is increased to 30 from 60:
   Find the line
   
   def close(self, timeout=30):  <<<<<< i to Insert change this value to 60
      '''Close untar stream and wait for process completion.
         Parameters:
            * timeout - the amount of time in seconds, to wait for output
                        or completion of the process.

   Save the file:
   Esc
:wq!
7. Move the modified LockerInstaller.py file back to its original location overwriting the original:
    mv /tmp/backup/LockerInstaller.py /lib64/python3.8/site-packages/vmware/esximage/Installer/LockerInstaller.py
8. Run the following command:
   vsish -e set /config/VisorFS/intOpts/VisorFSPristineTardisk 1
9. Retry remediation of host

Workaround 2:

1. After increasing the timeout in workaround 1,
   Manually install VMware tools by downloading offline installer bundle of ESXi that matches the version of the single image that is configured at the cluster level.
   
   

   esxcli software component apply -n VMware-VM-Tools:tools_version -d <pathtodepotzip>

   Example:
   esxcli software component apply -n VMware-VM-Tools:12.3.5.22544099-23305545 -d <pathtodepotzip>