ESXi upgrade Pre-check is failing on the SDDC Manager with "The validator 'bundle-validator' did not finish in 5400 seconds and was forcefully terminated"

Products

• VMware SDDC Manager

Issue/Introduction

• The upgrade timeout fails with the error: "The validator 'bundle-validator' did not finish in 5400 seconds and was forcefully terminated"
  • 
• You will not be able to fetch the error message: "The validator 'bundle-validator' did not finish in 5400 seconds and was forcefully terminated" in any logs 
• The ESXi host scan entity task on the vCenter UI would intially be stuck at 0%  but would take longer time to complete (approx 20+ hrs)
• In the  /var/log/vmware/vmware-updatemgr/vum-server.log, you will see multiple traces of baselines like the below: 

  • [#40] YYYY-MM-DDT12:54:27.740Z info vmware-vum-server[12968] [Originator@6876 sub=Activation.trace] [activationValidator 1131] Invoke done: integrity.ComplianceStatusManager.queryBaselineComplianceStatus

    [#40] --> Result:

    [#40] --> (integrity.BaselineComplianceStatus) [

    [#40] -->    (integrity.BaselineComplianceStatus) {

    [#40] -->       key = 1,

    [#40] -->       status = "Unknown"

    [#40] -->    },

    [#40] -->    (integrity.BaselineComplianceStatus) {

    [#40] -->       key = 2,

    [#40] -->       status = "NotCompliant"

    [#40] -->    },

    [#40] -->    (integrity.BaselineComplianceStatus) {

    [#40] -->       key = 64,

    [#40] -->       status = "NotCompliant"

    [#40] -->    }

    [#40] --> ]

    .

    .

    YYYY-MM-DDT12:54:27.787Z info vmware-vum-server[36694] [Originator@6876 sub=Activation.trace] [activationValidator 1064]

    --> ------------------------------------------------------

    --> Invoking queryBaselineGroupComplianceStatus on integrity.ComplianceStatusManager:Integrity.ComplianceStatusMgr

    --> Arg entity:

    --> 'vim.HostSystem:5cc74788-6808-4d20-bfa4-d35b40f9a3a3:host-3015'

    --> ------------------------------------------------------

    .

    .

    [#40] YYYY-MM-DDT12:54:27.802Z info vmware-vum-server[36694] [Originator@6876 sub=Activation.trace] [activationValidator 1131] Invoke done: integrity.ComplianceStatusManager.queryBaselineGroupComplianceStatus

    [#40] --> Result:

    [#40] --> (integrity.BaselineGroupComplianceStatus) [

    [#40] -->    (integrity.BaselineGroupComplianceStatus) {

    [#40] -->       key = 10,

    [#40] -->       status = "Compliant"

    [#40] -->    },

    [#40] -->    (integrity.BaselineGroupComplianceStatus) {

    [#40] -->       key = 51,

    [#40] -->       status = "NotCompliant"

    [#40] -->    }

    [#40] --> ]

Environment

SDDC Manager 5.2.1.2

Cause

• While upgrading the ESXi host from the SDDC Manager, the scanning of the host took 22hrs while the SDDC Manager ESXi upgrade timeout value is only for 4hrs due to which the pre-check failed with a timeout error
• The scanning took 22hrs because it was trying to scan multiple Baselines that were present 

Resolution

Before applying the steps below, take a backup or an offline-snapshot (in powered-off state) of the vCenter Server. If the vCenter Server is part of an ELM environment, take a snapshot or a backup of all vCenters within the ELM domain. Note all of the custom configurations within Update Manager - e.g. proxy settings, third party download URLs, customized baselines, etc. - before proceeding.

Reset the VUM Database as per the reference KB: Resetting the VMware Update Manager Database

Note:

• If the NSX-T is configured, make sure to perform the workaround suggested in the above KB for reset the Update Manager database if NSX-T is installed in a vLCM-enabled cluster (applicable for vCenter Server Appliance 7.0 U1 and beyond).
• Also, make sure the dl.broadcom.com URL's present in Lifecycle Manager should be taken as a backup before resetting the VUM database and added back once the DB reset is completed