Stateless host stuck at /vmw/rbd/host-register after applying new Image profile

Products

• VMware vSphere ESXi

Issue/Introduction

• After applying a new Image profile, a stateless ESXi host becomes stuck at /vmw/rbd/host-register... in an infinite loop during the boot process.
• The rbd-cgi.log file (located under /var/log/vmware/rbd/ in the vCenter Server Appliance) displays a repeating HTTP 503 error, indicating the service is unavailable:
  
  INFO:sslcert:Host has a moref "host-######", certpath: None
  INFO:director:--- "GET /vmw/rbd/host/5c64.../config?bootmac=00:AA:BB:AA:BB:CC HTTP/1.1" 503 -

Environment

 

• VMware vCenter Server

• VMware ESXi (Stateless / Auto Deploy)

 

Cause

The ESXi host reaches out to the Auto Deploy web server to request its boot configuration, but the backend Auto Deploy worker service (responsible for building the image cache) fails to respond, returning an HTTP 503 (Service Unavailable) error. Because the host receives this error, it waits, prints a dot on the screen, and retries indefinitely.

The log line immediately preceding the 503 error (certpath: None) indicates that Auto Deploy cannot find a valid certificate path for the host. This typically occurs when a host is remediated, or if vCenter certificates expire or change. Auto Deploy fails to authenticate the host's request and crashes the worker thread, resulting in the 503 error.

Resolution

To resolve this issue, you must restart the Auto Deploy services on the vCenter Server Appliance (VCSA) to clear the stuck worker thread.

1. Open an SSH session to the vCenter Server Appliance.

2. Run the following command to restart the Auto Deploy Waiter service:

service-control --restart vmware-rbd-watchdog

3. Once the service restarts successfully, reboot the ESXi host to force it to start a fresh PXE boot session.