Auto Deploy Cache Corruption When Using Multiple ESXi Images

Products

VMware vSphere ESXi

Issue/Introduction

Auto Deploy Cache Corruption When Using Multiple ESXi Images in vCenter 8.0 Environment

ESXi hosts managed by Auto Deploy fail to boot after 3-4 days of adding multiple ESXi images
Error messages similar to the following appear in /var/log/vmware/rbd/rbd-syslog.log:

Failed to repair the cache: Something went wrong while converting items to pxe profile:'NoneType' object has no attribute 'key'

Regular execution of Repair-DeployImageCache temporarily resolves the issue, but the problem recurs after 3-4 days
Auto Deploy service returns HTTP 5xx errors during the boot process.
/var/log/vmware/rbd/rbd-cgi.log contains: ERROR:pluginmaster:exception:rbdplugins.plainpxe.vmwGetGpxeConf -- 'VMware_bootbank_vmware-fdm_8.0.3-####' ... KeyError: 'VMware_bootbank_vmware-fdm_8.0.3-####'.
/var/log/vmware/rbd/rbd-cached.log shows frequent "flushing cached item" messages despite the UI indicating sufficient cache size.

Environment

vCenter Server 8.0.x
Multiple ESXi image profiles (mixing 7.x and 8.x)

Cause

The Auto Deploy cache size configuration file (autodeploy-setup.xml) may not update correctly when changed through the vCenter UI. If the default 2GB limit is reached, the service flushes vital VIBs from the cache, leading to a KeyError when a host attempts to request those files during PXE boot.

Resolution

Manually edit the Auto Deploy configuration file to increase the cache size:
a. SSH into the vCenter Server Appliance
b. Open the Auto Deploy setup configuration file using vi or another text editor:
```
vi /etc/vmware-rbd/autodeploy-setup.xml
```
c. Locate the <maxSize> parameter under <defaultValues> and change its value from the default (likely 2) to a higher value (8 or more is recommended):
```
<defaultValues>
    <port>6501</port>
    <portAdd>6502</portAdd>
    <maxSize>8</maxSize>
</defaultValues>
```
d. Save the file and exit the editor

e. Open the Auto Deploy props file using vi or another text editor:
```
vi /usr/lib/vmware-rbd/config/autodeploy-config.props
```
f. Locate the cachesize_GB value and change it from the default to the same value as you set above:
```
cachesize_GB=8
serviceport=6501
managementport=6502
loglevel=INFO
```
g. Save the file and exit the editor

h. Restart the Auto Deploy service:
```
service-control --restart rbd
```
i. Verify that the cache size has been updated correctly:
```
grep -rin /var/log/vmware/rbd -e "cacher starting"
```
j. The output should include a line similar to:
```
INFO:rbd_cached:cacher starting (/var/lib/rbd/cache, 8589934592)...
```
The number 8589934592 represents 8GB in bytes, confirming the increased cache size.
Alternatives to modifying the configuration file:
a. Use the vCenter UI to add ESXi images to Software Depots instead of using PowerCLI
b. Use only one ESXi version at a time in Auto Deploy via PowerCLI
c. Regularly run the Repair-DeployImageCache command if the above solutions cannot be implemented

Additional Information

The issue appears to be specifically related to how cache management occurs after an upgrade from vCenter 7.0.x to vCenter 8.0.x. The UI setting for cache size may not properly update the underlying configuration file.

The problem is more likely to occur when using both 7.x and 8.x images simultaneously via PowerCLI because:

ESXi 8.0 images are larger than 7.0 images due to unified depots
PowerCLI-created rules store VIBs differently than UI-created Software Depot entries

For further troubleshooting of Auto Deploy service responsiveness, see KB 374630. If technical assistance is required, see Contact Support.