Manually Patching an ESXi Host from the CLI failed with " Stateless error".

Products

VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

Manually Patching an ESXi Host from the CLI failed with " Stateless error".

[StatelessError]

The transaction is not supported: VIB VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.1.39.11675023 cannot be live installed. VIB VMware_bootbank_esx-base_6.7.0-1.39.11675023 cannot be live installed. VIB VMware_bootbank_esx-update_6.7.0-1.39.11675023 cannot be live installed. VIB VMware_bootbank_vsan_6.7.0-1.39.11399593 cannot be live installed. VIB VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-8vmw.670.0.0.8169922 cannot be removed live. VIB VMware_bootbank_esx-update_6.7.0-1.28.10302608 cannot be removed live. VIB VMware_bootbank_esx-base_6.7.0-1.28.10302608 cannot be removed live. VIB VMware_bootbank_vsan_6.7.0-1.28.10290435 cannot be removed live.

/var/log/boot logs reports below error
YYYY-MM-DDTHH:MM:31.282Z cpu4:2097604)Jumpstart plugin tag-boot-bank activation failed: tag-boot-bank->start() failed: exited with code 1
Bootbank points to tmp

root@Esxi_fqdn :~ ] 1s -al /
otal 1189
lrwxr-xI-x 1 root root	512 Apr 1 09:19
---- T 1 root root	193 Jun 16 2024 .fencryption. info
rWXI-XI-X 1 root TOOt	512 Apr 1 09:19 .
rw -- 1 root root	9 Apr 1 09:19 .ash history
r -- r -- r -- 1 root root	20 Apr 28 2023 .mtoolsrc
rw-rw-rwT 1 root root	0 Apr 1 09:14 COVData. json
rw-rw-rwT 1 root root	0 Apr 1 09:14 CSUTData. json
rw-rw-IwT 1 root root	1 09:14 CSUTSettingsData. json 0 Apr
IW-IW-IWI 1 root root	1 09:14 CSUTiLODetails. json 0 Apr
IW-IW-IWI 1 root root	1 09:14 CSystemInventory. json 0 Apr
IW-IW-IWI 1 root root	1 09:14 backup_cfg. dat 0 Apr
rWXI-XI-X root root	1 09:14 bin 512 Apr
IWXIWXIWX root 1 root	1 09:14 bootbank -> /tmp/ bootbank81g54eas 22 Apr
r -- r -- r -- root root	2023 bootpart.gz 430381 Apr 28
r -- r -- r -- 1 root root	2023 bootpart4kn.gz 307741 Apr 28
rwxr-xr-x 16 root root	1 09:19 dev 512 Apr
rwxr-xr-x 1 root root	1 09:14 etc 512 Apr
rwxr-xr-x 1 root root	1 09:13 include 512 Apr
lrwxr-xr-x 1 root root	1 09:13 lib 512 Apr
rWXI-XI-X root root	1 09:13 lib64 512 Apr
rW-I -- I -- root 1 root	113871 Apr 1 09:13 local.tgz
r -- r -- r -- 1 root root	126976 Jun 16 2024 local.tqz.ve
rwxrwxrwx 1 root root	1 09:14 locker -> /var/lib/vmware/osdata/locker 29 Apr
rwxr-xr-x 1 root root	1 09:13 opt. 512 Apr
rwxr-xr-x 1 root root	1 09:19 proc 131072 Apr
rwxrwxrwx IWXIWXIWX 1 root 1 root root root	1 09:14 productLocker -> /locker/packages/vmtoolsRepo 2023 sbin -> /bin 28 Apr 4 Apr 28
IWXIWXIWX IWXIWXIWX 1 root 1 root root root	22 Apr 28 Apr 1 09:14 scratch -> /var/lib/vmware/osdata 1 09:14 store -> /var/lib/vmware/osdata/store
rW-IW-IWT rw-rw-rwI 1 root root root root	0 Apr 0 Apr 1 09:14 sut.pdb 1 09:14 sut aes
rwxr-xr-x rwxr-xr-x 1 root 1 root root root	512 Apr 512 Apr 1 09:13 tardisks 1 09:13 tardisks. noauto
rwxrwxrwt 1 root root	512 Apr 1 09:15 tmp
rwxr-xr-x lrwxr-xr-x 1 root root root root	512 Apr 512 Apr 1 09:13 usr 1 09:14 var

Environment

Esx 7.x

Esx 8.x

Cause

This is a common issue where certain kernel modules fail to unload.

And if /Bootbank is pointing to /TMP Directory.

Resolution

Make sure the host is in maintenance mode.
reboot the ESXI host.
then run the command again.

If not worked then need to check and confirmed weather /Bootbank is pointing to /TMP directory. if yes then need to analyze it accordingly and it should point to the /vmfs/volume Datastore.

Reboot the ESXi host.
During the pre-boot splash screen, press SHIFT+O to modify the boot option line.
In the resulting screen, move to the end of the boot line.
Add the following to the same line:

devListStabilityCount=30

For example:

Press the enter key to resume boot.
Login to the ESXi shell or SSH after the system has rebooted
Run the following command:

ls -al /

The output will list the links for bootbank and altbootbank. Confirm the path shows as /vmfs/volumes/UUID for both.

Note: If the path remains as /tmp/, you must reboot the ESXi host and increase the devListStabilityCount value in the boot option line (see earlier steps). Do not proceed further with the instructions until this has been corrected.

Once the bootbank/altbootbank directories are directed to /vmfs/volumes/UUID, navigate to the bootbank directory:

cd /bootbank/

Make a backup of the boot.cfg

cp boot.cfg boot.cfg.bak

Edit boot.cfg to add the following setting to the line beginning with kernelopt

devListStabilityCount= the value you set in boot option line

Before:

After:

Save the changes, and reboot the ESXi host to apply the workaround.