Auto Deploy rule changes are not applied on host boots, or new rule creations fail.

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

When changing a rule in Auto Deploy, the hosts do not pick up the changes to the rule on the next boot.

When creating a new rule in Auto Deploy, a "Operation failed!" message box appears, and the task fails with an error of "fault.com.vmware.rbd.taskerror.summary".

ESXi hosts will fail to PXE boot.

In the rbd-vc-monitor.log, you see the following error repeating over and over:

2019-09-20T12:11:59.532 [31266]INFO:DepotCollection:Could not download from depot at http://<VC_FQDN_OR_IP>:80/vSphere-HA-depot/index.xml, skipping (('http://<VC_FQDN_OR_IP>:80/vSphere-HA-depot/index.xml', '', 'HTTP Error 502: Bad Gateway'))
2019-09-20T12:11:59.533 [31266]ERROR:vibutil:encountered errors while connecting to depots: [MetadataDownloadError('http://<VC_FQDN_OR_IP>:80/vSphere-HA-depot/index.xml', '', "Could not download from depot at http://<VC_FQDN_OR_IP>:80/vSphere-HA-depot/index.xml, skipping (('http://<VC_FQDN_OR_IP>:80/vSphere-HA-depot/index.xml', '', 'HTTP Error 502: Bad Gateway'))")]
2019-09-20T12:11:59.533 [31266]ERROR:pluginmaster:exception:rbdplugins.plainpxe.vmwMonitorInit -- ('', "No VIB matching VIB search specification 'VMware:vmware-fdm'.")

This log shows the vc-monitor crashing and exiting, then starting again and attempting to download the FDM VIB from the depot on the vCenter Server.

Environment

VMware vCenter Server 6.x
VMware vCenter Server 7.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 6.x

Cause

There are typically two causes of this issue:

A proxy configuration which does not allow the traffic to pass to the vCenter and download the VIB
Problems communicating to the vCenter via the FQDN/IP. (eg DNS resolution)

The vc-monitor component of Auto Deploy is responsible for updating the associations from rules to the Auto Deploy database. Because it continually crashes, any changes to existing rules or new rules cannot be propagated to the database, causing hosts to fail to PXE boot, or to boot unexpectedly to the old rule values.

Resolution

Currently, there is no resolution.

Workaround:
To work around this issue, alter the Auto Deploy database and change the value of the vCenter Server address from the FQDN to the internal loopback - 127.0.0.1.

1. Stop the Auto Deploy service:

service-control --stop vmware-rbd-watchdog

2. Take a backup of the Auto Deploy database, located here:

/var/lib/rbd/db

3. Run the following command to open the Auto Deploy database in sqlite:

sqlite3 /var/lib/rbd/db

4. Verify the current values in the vc_servers table by running the following query in sqlite:

select * from vc_servers;

5. You should see output similar to the following, where the vCenter shows either FQDN or the IP address of itself:

1|vcenter.example.com|80|e52ab74c-3950-4119-b268-985cede8934e|8088|

6. Run the following query to update the value of the vCenter Server address to use the loopback:

update vc_servers set addr="127.0.0.1";

7. Check that the value has changed by running the following query again:

select * from vc_servers;

8. You should see output similar to the following:

1|127.0.0.1|80|e52ab74c-3950-4119-b268-985cede8934e|8088|

9. To exit sqlite, use the following command:

.quit

10. Start the Auto Deploy service.

service-control --start vmware-rbd-watchdog

11. Use the above commands to connect back to the Auto Deploy database and verify that the vc_servers table still shows the correct address. Also, run the following query to verify that the FDM depot address has also been updated:

select * from vib_depot;

12. You should see output similar to the following:

__vc|http://127.0.0.1:80/vSphere-HA-depot|unknown

13. Attempt to create/change rules, and ensure a host will boot correctly. Review the rbd-vc-monitor.log file to ensure the monitor is no longer crashing/exiting over and over.