Stateless Autodeploy hosts fail to boot after vCenter upgrade

Products

• VMware vCenter Server

Issue/Introduction

To fix the autodeploy ESXi boot error.

Symptoms:

• After a vCSA upgrade, the autodeploy service may stop provisioning hosts
• HTTP 5xx Server Error  is seen when booting hosts with Auto Deploy.
• In the vCenter , rbd-cgi.log you see similar entries as below

 2021-05-20T08:45:40.670 [34968]ERROR:miscutil:Error: Could not obfuscate string rc: 1 
error:b'bad decrypt\n140412449614592:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:563:\n'
2021-05-20T08:45:40.671 [34968]ERROR:pluginmaster:exception:rbdplugins.hostprofile.vmwGetGpxeConf -- pickle data was truncated
Traceback (most recent call last):
  File "/build/mts/release/bora-17920184/BOD/vcenter-autodeploy/linx64/release/9c6a169/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/utils/pluginmaster.py", line 236, in _curry
  File "/build/mts/release/bora-17920184/BOD/vcenter-autodeploy/linx64/release/9c6a169/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbdplugins/hostprofile.py", line 812, in vmwGetGpxeConf
_pickle.UnpicklingError: pickle data was truncated

Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x

Cause

Due to openssl procedure explicitly set -md=256 which causes decryption to fail.

Resolution

This is a known issue and currently there is no resolution.

Workaround:
The current workaround is to manually identify and delete the "corrupted" cached items. To do this follow the below steps

1. Login to VCSA using SSH  and run command
find /storage/autodeploy/cache/ -name .hostprofile-secrets
   - it lists the exact location of .hostprofile-secrets
   It looks like this :
root@vcenter [ ~ ]# find /storage/autodeploy/cache/ -name .hostprofile-secrets
/storage/autodeploy/cache/5b/0da9cb7625f24b478819cafc323f56/.hostprofile-secrets
/storage/autodeploy/cache/ac/061f20eb3b08629545ad9f5b9e3705/.hostprofile-secrets
/storage/autodeploy/cache/b3/12b935391d1eb19fa38f4498f08685/.hostprofile-secrets

2.For each .hostprofile-secrets location delete the entire dir
  - example : >> [ ~ ]# rm -rf /storage/autodeploy/cache/5b/0da9cb7625f24b478819cafc323f56/

3. Boot the host again.

Additional Information

Impact/Risks:
None