After upgrading from vCenter Server 6.5 to vCenter Server 7.0, ESXi hosts provisioned through Auto Deploy fail to boot

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:
After upgrading to vCenter Server 7.0, ESXi hosts in an Auto Deploy environment may fail to boot.

The following error is displayed during the boot attempt:
 

Could not boot image: HTTP 5xx Server Error

You see the following error block in the rbd-cgi.log (bold added for emphasis):

2020-11-19T16:07:32.571 [12120]ERROR:director:CGI handler failed -- '<=' not supported between instances of 'str' and 'bytes'
Traceback (most recent call last):
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/waiter/cgi/director.py", line 192, in do_GET
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/waiter/cgi/director.py", line 126, in handler
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/waiter/cgi/supported_handlers.py", line 1101, in hostPxeConfig
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/waiter/pxe_context.py", line 67, in __init__
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/waiter/pxe_context.py", line 227, in _matchingRulesForHost
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/compiler/rule_compiler.py", line 141, in matchingRulesForHost
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/compiler/quicksect.py", line 29, in intersectPoint
File "/build/mts/release/bora-17004913/BOD/vcenter-autodeploy/linx64/release/4a03407/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/compiler/quicksect.py", line 117, in intersectPoint
TypeError: '<=' not supported between instances of 'str' and 'bytes'

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server 7.0.x

Cause

This is caused by an upgrade in the version of Python running on the appliance.  Auto Deploy caches some information in order to speed up processes and this cached data is interpreted differently by the new Python version.  Because of this, comparisons between objects fail.

While this issue has been observed in an upgrade from vCenter Server 6.5 to vCenter Server 7.0, it may also affect an upgrade from vCenter Server 6.5 to vCenter Server 6.7 as an updated version of Python is also present in vCenter Server 6.7.

Resolution

There is currently no resolution.

Workaround:
To workaround the issue, disable and then re-enable each rule in Auto Deploy.  By doing this, cached values will be cleared and will no longer cause a problem.  This only needs to be done once for each rule.