ESXi host upgrade fails with "The host returns esxupdate error code:15. The package manager transaction is not successful"

Products

• VMware vSphere ESXi

Issue/Introduction

To resolve this error when remediating the ESXi 5.x/6.x host using VUM or upgrading the host from the command line.

Symptoms:

• Remediating ESXi 5.x or 6.x hosts using VUM fails with error code:15
• You see the below error message 

          The host returns esxupdate error code:15. The package manager transaction is not successful. Check the Update Manager log files and esxupdate log files for more details.

• In the /var/log/esxupdate.log file, you see entries similar to:

2018-12-04T20:29:45Z esxupdate: 205640: LockerInstaller: WARNING: There was an error in cleaning up product locker: [Errno 5] Input/output error: '/locker/packages/6.5.0/vmtools'
2018-12-04T20:29:46Z esxupdate: 205640: esxupdate: ERROR: vmware.esximage.Errors.InstallationError: ('VMware_locker_tools-light_6.5.0-0.23.5969300', '[Errno32] Broken pipe')

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

• It also observed when cd to /locker/packages/6.5.0 fails you may see the input/output error.

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Resolution

To resolve this issue, follow below-provided steps, please do not skip

1. Run ls -ltrh and identify the device UUID that is being used for /store.

           lrwxrwxrwx    1 root     root      49 Nov 26 12:06 store -> /vmfs/volumes/534c2849-########-####-##########4c

        2. Identify the disk device by using the below command:  
      vmkfstools -P /vmfs/volumes/vmfs/volumes/534c2849-########-####-##########4c

Example:

[root@hostname:~] vmkfstools -P /vmfs/volumes/572398ac-########-####-#########a0
vfat-0.04 file system spanning 1 partitions.
File system label (if any):
Mode: private
Capacity 299712512 (36586 file blocks * 8192), 299089920 (36510 blocks) avail, max supported file size 0
UUID: 572398ac-########-####-##########a0
Partitions spanned (on "disks"):
        mpx.vmhba32:C0:T0:L0:8
Is Native Snapshot Capable: NO

         3. Re-format the device disk from above step with vfat filesystem (this will create a new UUID)  vmkfstools -C vfat /dev/disks/mpx.vmhba32:C0:T0:L0:8 

[root@hostname:~] vmkfstools -C vfat /dev/disks/mpx.vmhba32:C0:T0:L0:8
create fs deviceName:'/dev/disks/mpx.vmhba32:C0:T0:L0:8', fsShortName:'vfat', fsName:'(null)'
deviceFullPath:/dev/disks/mpx.vmhba32:C0:T0:L0:8 deviceFile:mpx.vmhba32:C0:T0:L0:8
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vfat file system on "mpx.vmhba32:C0:T0:L0:8" with blockSize 1048576 and volume label "none".
The filesystem was created but mount failed on device "mpx.vmhba32:C0:T0:L0:8".: Not found.
Successfully created new volume: 5bec6252-########-####-##########a0

[root@hostname:~] vmkfstools -P /vmfs/volumes/572398ac-########-####-#########a0
vfat-0.04 file system spanning 1 partitions.
File system label (if any):
Mode: private
Capacity 299712512 (36586 file blocks * 8192), 299089920 (36510 blocks) avail, max supported file size 0
UUID: 572398ac-########-####-##########a0
Partitions spanned (on "disks"):
        naa.644a8420135982001e83017d0ce370f6:8
Is Native Snapshot Capable: NO


        4.  Re-create symlink with new UUID:[root@hostname:~] ln -snf /vmfs/volumes/<new-UUID from above step> store
​​
        5.  Copy packages, var under locker directory from working host to non-working host. (Same version, build can be different)
       
        6.  Retry the scan/remediation.