In place vCenter upgrade to 9.1 fails at VCENTER UPGRADE PRECHECK

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

When upgrading vCenter form 9.0 to 9.1 it fails on VCENTER UPGRDE PRECHECK with the following error in the UI:

VCF LCM Service

Token: ######
Occurrence: DD/MM/YYYY HH:MMReason is: InternalServerError
(com.vmware.vapi.std.errors.intemal_server_error)
=> { messages = [LocalizableMessage
(com.vmware.vapi.std.localizable_message) => {
id = vapi.data.invalid, defaultMessage = Invalid
Type, expected String, instead got NoneType,
args = [String, NoneType], params = <null>,
localized = <null> }]. data = <null>, errorType =
INTERNAL_SERVER_ERROR ].

In the /var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDTHH:MM:SS.000+0000 DEBUG [vcf_lcm,] [c.v.e.s.l.s.impl.UpgradeServiceImpl,http-nio-127.0.0.1-7400-exec-7] Start collecting the cluster ids for all Cluster level
 Upgrades
YYYY-MM-DDTHH:MM:SS.000+0000 DEBUG [vcf_lcm,] [c.v.evo.sddc.lcm.model.task.SubTask,http-nio-127.0.0.1-7400-exec-7] Upgrade error occured: Reason is: InternalServerError
(com.vmware.vapi.std.errors.internal_server_error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = vapi.data.invalid,
    defaultMessage = Invalid Type, expected String, instead got NoneType,
    args = [String, NoneType],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = INTERNAL_SERVER_ERROR
}.
Reference token #####

Environment

VMware Cloud Foundation 9.1

VMware vSphere Foundation 9.1

VMware vCenter Server 9.0.x

VMware vCenter Server 9.1

Cause

The SRP bind of the DCAccount is failing here when the dir-cli command is triggered during pre-check to fetch the VMDir state.
The DCAccount is locked because of number of failed attempts to bind with “incorrect” password several times.

The following log entries are observed:

/var/log/vmware/applmgmt/PatchRunner.log:

YYYY-MM-DDTHH:MM:SS.000Z ERROR applmgmt-patch-runner 133390 [vc@4413 ComponentHook="vmdir:CollectRequirements"] dir-cli failed. Error 382312694: Access denied, reason = rpc_s_auth_method (0x16c9a0f6).

/var/log/vmware/vmdird/vmdird.log:

vmdird.log:YYYY-MM-DDTHH:MM:SS.000Z:t@:ERROR: VdirPasswordFailEvent from user(cn=vmware.com,ou=domain controllers,dc=vsphere,dc=local), error(0)()
vmdird.log:YYYY-MM-DDTHH:MM:SS.000Z:t@:ERROR: Srv_rpc_srp_verifier_verify_session failed, status (382312692)
vmdird.log:YYYY-MM-DDTHH:MM:SS.000Z:t@:ERROR: SRP authentication failure - User: vmware.com@vsphere.local

There are multiple references of bind failure as seen below.

vmdird.log:YYYY-MM-DDTHH:MM:SS.000Z:t@:ERROR: VdirPasswordFailEvent from user(cn=vmware.com,ou=domain controllers,dc=vsphere,dc=local), error(0)()
vmdird.log:YYYY-MM-DDTHH:MM:SS.000Z:t@:ERROR: VdirPasswordFailEvent from user(cn=vmware.com,ou=domain controllers,dc=vsphere,dc=local), error(0)()
vmdird.log:YYYY-MM-DDTHH:MM:SS.000Z:t@:ERROR: VdirPasswordFailEvent from user(cn=vmware-.com,ou=domain controllers,dc=vsphere,dc=local), error(0)()

Resolution

Please refer the KB https://knowledge.broadcom.com/external/article/316608 to reset the DCAccount/Machine Account credentials and retry the upgrade.