Esxi Host remediation error : "Host remediation operation could not be started because no hosts are available for remediation."

Products

• VMware vSphere ESXi

Issue/Introduction

While doing the remediation on ESXi, getting error: "Host remediation operation could not be started because no hosts are available for remediation". 

on vCenter /var/log/vmware/vpxd/vpxd.log

2024-07-10T12:29:37.087Z info vpxd[08477] [Originator@6876 sub=CryptoManager opID=lj4y35y1-6740913-auto-40hbm-h5:71473201-24-SWI-52b8fbd4] Remediation completed for host [vim.HostSystem:host-####,esxi-fqdn]
2024-07-10T12:34:59.570Z info vpxd[08712] [Originator@6876 sub=CryptoManager opID=lj4y35y1-6741728-auto-40hy9-h5:71473546-e8-SWI-1c8efd43] Remediation completed for host [vim.HostSystem:host-####,esxi-fqdn]

2024-07-10T13:04:08.244Z warning vpxd[06541] [Originator@6876 sub=vmomi.soapStub[983682] opID=lj4y35y1-6745145-auto-40kl7-h5:71474281-c-01] SOAP request returned HTTP failure; <SSL
(<io_obj p:0x00007f4e80070c58, h:92, <TCP '##:##:##:## : 40010'>, <TCP '##:##:##:## : 443'>>), /vpxa>, method: generateHostConfigTaskSpec; code: 500(Internal Server Error)


2024-07-10T13:04:08.245Z error vpxd[06541] [Originator@6876 sub=moHostProfileMgr opID=lj4y35y1-6745145-auto-40kl7-h5:71474281-c-01] [GenerateHostConfigTaskSpecInt1] Failed to gener
ate task configuration for host [vim.HostSystem:host-99227] with fault [Fault cause: vmodl.fault.SystemError
--> ]:[Received SOAP response fault from [<SSL(<io_obj p:0x00007f4e80070c58, h:92, <TCP '##:##:##:##: 40010'>, <TCP '##:##:##:## : 443'>>), /vpxa>]: generateHostConfigTaskSpec
--> Received SOAP response fault from [<<io_obj p:0x000000e0c13dee88, h:29, <TCP '127.0.0.1 : 19129'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: generateHostConfigTaskSpec
--> A general system error occurred: ]

 

/var/run/log/syslog.log on ESXi host

 

2024-07-10T13:04:07Z Host Profiles[2106122 opID=lj4y35y1-6745145-auto-40kl7-h5: 71474281-c-01-13-2e68]:WARNING: excObj has a faultMessage^@
2024-07-10T13:04:07Z Host Profiles[2106122 opID=lj4y35y1-6745145-auto-40kl7-h5: 71474281-c-01-13-2e68]:INFO: Cleaned up Host Configuration^@
2024-07-10T13:04:07Z Host Profiles[2106122 opID=lj4y35y1-6745145-auto-40kl7-h5: 71474281-c-01-13-2e68]:ERROR: Exception while generating task config spec. Exception: (vmodl.fault.SystemError) {    dynamicType = <unset>,    dynamicProperty = (vmodl.DynamicProperty) [],    msg = <unset>,    faultCause = <unset>,    faultMessage = (vmodl.LocalizableMessage) [       (LocalizableMessageWithPath) {          dynamicType = <unset>,          dynamicProperty = (vmodl.DynamicProperty) [],          key = 'com.vmware.vim.profile.caching.CannotPreserveVmfsEx',          arg = (vmodl.KeyAnyValue) [             (vmodl.KeyAnyValue) {                dynamicType = <unset>,                dynamicProperty = (vmodl.DynamicProperty) [],                key = 'disk',                value = 'naa.6000006,...,naa.6000006'             }          ],          message = "A VMFS volume was found on 'naa.6000006,...,naa.6000006', but we are unable to preserve it

 

 

You might notice esxi host is booting from /tmp directory : 

 

bootbank -> /tmp/_bootbank

 

Environment

VMware ESXi 

Cause

This issue happens if the host does not have any disk which is eligible for stateless caching as all the LUN's have VMFS partitions.

 

Resolution

Add a new empty disk to ESXi host which can be used for Stateless caching.

Refer :

Introduction to Stateless Caching and Stateful Installs

Tagging the Stateless Caching Disk using Autodeploy displays an inaccurate Is Local: value

 

Additional Information

ESXi was booting from Auto deploy (Stateless Cache)

Host profile was applied.