Auto deploy is not booting hosts and web ui is not responding

Products

VMware vSphere ESXi VMware vCenter Server VMware vCenter Server 8.0 VMware vSphere ESXi 8.0

Issue/Introduction

When booting a host using Auto Deploy, you see messages such as:

Could not boot: Connection Reset

Could not boot image: Connection Reset

The host is attempting to network boot using VMware Auto Deploy. However, script bundle is missing from the autodeploy cache.

Example host bootup screen with error messages as above:

Environment

VMware vCenter Server 8.x

VMware vSphere ESXi 8.x

Cause

In the rbd-syslog.log file you see the following errors:

err rbd  [2854759]director:ERROR:Failed to check the VCHA value
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cgi/director.py", line 277, in _isVCHAConfigured
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/vc_servers.py", line 149, in serviceInstance
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/vc_servers.py", line 78, in connect
Exception: No default VC configured
err rbd  [3100814]networkutil:ERROR:127.0.0.1 : hostAddr is either loopback or link-local

Also can see errors related to no profiles found:

[611717:CacheWorker-0]ERROR:pxe_profile_util:Something went wrong while converting items to pxe profile:No profiles found with the given name
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vmodl_impl/pxe_profile_util.py", line 151, in ConvertToPxeProfile
rbd.vmodl_impl.pxe_profile_util.ProfileMiss: No profiles found with the given name
[611717:CacheWorker-0]ERROR:pxe_profile_util:Failed to repair the cache: Something went wrong while converting items to pxe profile:No profiles found with the given name
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vmodl_impl/pxe_profile_util.py", line 151, in ConvertToPxeProfile
rbd.vmodl_impl.pxe_profile_util.ProfileMiss: No profiles found with the given name

Resolution

The cache files stored within /storage/autodeploy/cache/ on the vCenter appliance for a host are no longer valid and need to be reset. This is done by removing the cache file directories and the auto deploy database and then re-registering auto deploy service with vCenter. This recreates a new database for auto deploy and new cache files for the hosts.

Note: This will remove all hosts from within the Auto Deploy database. As a result before a host is rebooted it will need to be removed from inventory within vCenter so that the new rule(s) can be picked up. Only perform this step if Auto Deploy cannot be repaired or restored by other troubleshooting methods.

Workaround:

Create snapshot of vCenter or verify a recent backup exists for VCSA.

Stop autodeploy (vmware-rbd-watchdog)

service-control --stop vmware-rbd-watchdog

Unregister with autodeploy-register.

autodeploy-register -U -a vCenterFQDN -u [email protected] -w password -p 80

Delete files in /etc/vmware-rbd/db and /storage/autodeploy/cache/* (files and folders)
- ```
rm -rf /etc/vmware-rbd/db/*
```
- ```
rm -rf /storage/autodeploy/cache/*
```

Re-register auto deploy with autodeploy-register.

autodeploy-register -R -a vCenterFQDN -u [email protected] -w password -p 80

Start vmware-rbd-watchdog service.

service-control --start vmware-rbd-watchdog

Remove any software depots within vSphere Client.

In Auto Deploy > Software Depots > Select Depot > Click Remove

Add New Software Depot(s) by importing new images.
Create Deploy Rules for the hosts. Verify new rules are active.
Reboot an ESXi host configured for a valid deploy rule and verify bootup successful.
- Note: Host should be removed from inventory within vCenter before rebooting, otherwise image will fail to be found to boot host.