Stateless Autodeploy hosts fail to boot after vCenter upgrade
book
Article ID: 345223
calendar_today
Updated On:
Products
VMware vCenter Server
Issue/Introduction
To fix the autodeploy ESXi boot error.
Symptoms:
After a vCSA upgrade, the autodeploy service may stop provisioning hosts
HTTP 5xx Server Error is seen when booting hosts with Auto Deploy.
In the vCenter , rbd-cgi.log you see similar entries as below
2021-05-20T08:45:40.670 [34968]ERROR:miscutil:Error: Could not obfuscate string rc: 1 error:b'bad decrypt\n140412449614592:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:563:\n' 2021-05-20T08:45:40.671 [34968]ERROR:pluginmaster:exception:rbdplugins.hostprofile.vmwGetGpxeConf -- pickle data was truncated Traceback (most recent call last): File "/build/mts/release/bora-17920184/BOD/vcenter-autodeploy/linx64/release/9c6a169/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbd/utils/pluginmaster.py", line 236, in _curry File "/build/mts/release/bora-17920184/BOD/vcenter-autodeploy/linx64/release/9c6a169/build/bora/build/build/autodeploy-compiled/release/linux64/pyc/rbdplugins/hostprofile.py", line 812, in vmwGetGpxeConf _pickle.UnpicklingError: pickle data was truncated
Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Environment
VMware vCenter Server 6.7.x VMware vCenter Server 7.0.x
Cause
Due to openssl procedure explicitly set -md=256 which causes decryption to fail.
Resolution
This is a known issue and currently there is no resolution.
Workaround: The current workaround is to manually identify and delete the "corrupted" cached items. To do this follow the below steps
1. Login to VCSA using SSH and run command find /storage/autodeploy/cache/ -name .hostprofile-secrets - it lists the exact location of .hostprofile-secrets It looks like this : root@vcenter [ ~ ]# find /storage/autodeploy/cache/ -name .hostprofile-secrets /storage/autodeploy/cache/5b/0da9cb7625f24b478819cafc323f56/.hostprofile-secrets /storage/autodeploy/cache/ac/061f20eb3b08629545ad9f5b9e3705/.hostprofile-secrets /storage/autodeploy/cache/b3/12b935391d1eb19fa38f4498f08685/.hostprofile-secrets
2.For each .hostprofile-secrets location delete the entire dir - example : >> [ ~ ]# rm -rf /storage/autodeploy/cache/5b/0da9cb7625f24b478819cafc323f56/