Host profile fails with error "Host profile apply failed with error" through vSphere Auto Deploy when using vSphere Replication Appliance and Stateless Cache to USB.

Products

• VMware Live Recovery
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When a host profile containing distributed switch and stateless cache to USB configurations is used in vSphere Auto Deloy and vSphere Replication is deployed to this environment, "Host profile apply failed with error: ." message may happen after PXE boot.
• In /var/log/vmware/vpxd/vpxd.log, you can see similar to:

2022-12-06T07:20:35.140Z error vpxd[04298] [Originator@6876 sub=moHostProfileMgr opID=1c20dea4-02] [ReapplySingleHostInt1]: Reapply [vim.HostSystem:host-2392,esx-autodeploy.local] failed: Fault cause: vim.fault.InvalidState
--> ; Host is left in maintenance mode

• In /var/run/log/syslog.log, you can see similar to:

ESXi host detects correct caches in Postboot.
2022-12-06T07:19:26Z Host Profiles[2102168 opID=MainThread]: INFO: Postboot: USB already has correct stateful install or stateless caching on it^@
However, "The host's current cache is not equivalent to the booted image." is detected in reconnecting from vCenter Server if REAPPLY_REQUIRED is raised by DvsProfile.
2022-12-06T07:19:57Z Host Profiles[2103222 opID=1c20dea4-02-35-e2cf]: INFO: Returning Host Profile Manager state: (vmodl.KeyAnyValue) [  (vmodl.KeyAnyValue) {    dynamicType = <unset>,    dynamicProperty = (vmodl.DynamicProperty) [],    key = 'REAPPLY_REQUIRED',    value = (str) [     'DvsProfile'    ]  } ]
2022-12-06T07:20:23Z Host Profiles[2103509 opID=1c20dea4-02-fa-e331]: INFO: User chose for stateless caching to a USB disk.^@
2022-12-06T07:20:30Z Host Profiles[2103509 opID=1c20dea4-02-fa-e331]: INFO: System image caching compliance check result: [(LocalizableMessageWithPath) {  dynamicType = <unset>,  dynamicProperty = (vmodl.DynamicProperty) [],  key = 'com.vmware.vim.profile.caching.stateless.caching.esx.unmatch.image',  arg = (vmodl.KeyAnyValue) [],  message = "The host's current cache is not equivalent to the booted image." }].^@
2022-12-06T07:20:30Z Host Profiles[2103509 opID=1c20dea4-02-fa-e331]: WARNING: Task list for profile CachingProfile will require reboot if applied^@
After this, caching profile requires reboot.
2022-12-06T07:20:33Z Host Profiles[2103509 opID=1c20dea4-02-fa-e331]: INFO: Applying following profiles {'systemCache_caching_CachingProfile'} will require a reboot^@

• In /var/run/log/esxupdate.log, you can see similar to:

VIBs for vSphere Replication are installed between Postboot and reconnecting from vCenter Server.
2022-12-06T07:19:58Z esxupdate: 2103270: imageprofile: INFO: Adding VIB VMware_bootbank_vr2c-firewall_8.2.0.6324-14706510 to ImageProfile NEW-IMAGE-6.7-13473784-v470-1 _update^@
2022-12-06T07:20:05Z esxupdate: 2103924: imageprofile: INFO: Adding VIB VMware_bootbank_vmware-hbr-agent_6.8.6-1.1.14221959 to ImageProfile (Updated) NEW-IMAGE-6.7-13473784-v470-1 _update^@

Environment

VMware vSphere Replication 8.x
VMware vSphere ESXi 6.7
VMware vCenter Server 6.7.x

Cause

If new VIBs are installed after Postboot, a system cache on USB (which doesn't have the newly installed VIBs) doesn't match the booted image (which included the newly added VIBs).

Thus, the task generation step in the reapply operation generated stateless USB task. But, this stateless USB task only can be done at Postboot step.

As a result, reboot requirement is needed and an error is shown in vCenter Server.


 

Resolution

This is a known issue if installing other VIBs such as vSphere Replication VIBs happens between Postboot step and reapplying steps.

Workaround:

To workaround the issue, please follow one of the below mentioned steps:

• Add these other VIBs to an image profile for Auto Deploy. For more information about adding vSphere replication VIBs, see vSphere replication 8.2. ESXi unable to download hbr-agent.vib "VibDownloadError".

For example, while using vSphere Replication, download vr2c-firewall and vmware-hbr-agent VIBs from vSphere Replication Appliance and add these two VIBs to an image profile for vSphere Auto Deploy.

This should contain URLs https://<vSphere Replication IP>:8043/vib/vr2c-firewall.vib and https://<VR IP >:8043/vib/vmware-hbr-agent.vib.
OR from the above KB, these VIBs can be got directly from vSphere Replication Appliance by using SCP.

• Prevent ESXi host from installing these VIBs automatically.

1. Disable automatic installation by using hms-configtool.

/opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-vr2c-vib=false
/opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-hbragent-vib=false

2. Restart hms service.

service hms restart