vSAN HCL check continually fails to verify firmware version during Workload Domain pre-check

Products

• VMware Cloud Foundation

Issue/Introduction

To explain and work around issue with 4.x upgrade and vSAN checks from VMware Cloud Foundation's SDDC manager.

Symptoms:
Observed during upgrade process for 4.x versions of VMware Cloud Foundation:

VSAN HEALTH HCL INFO HEALTH

Description    Checks for VSAN HCL health.
Start Time    Date time stamp
End Time    Date time stamp

Health Status    RED
Error Description    VSAN Health: VSAN HCL Info: The controller device vmhba1:Broadcom 12G SAS HBA(16) that is running with driver <driver version for the HBA> shows up on the HCL, and is supported for the release of ESX in-use. But, the firmware version detected is not supported.  Impact    High: Do not perform ESXi upgrade without this addressing issue. Remediation    The Controller is on vSAN HCL and the Controller is supported for the currently running ESXi release.

It failed to further verify whether the firmware version in use is installed drivers.
If the firmware is not listed in the VCG for this device and ESXi release, the vSAN environment may be at risk.
This is important as firmware play a critical role in the stability and integrity of vSAN. Vendors often update their firmware to address critical bugs. In such cases, VMware may revoke the the certification status of an old firmware and only support the new version of the firmware. Hence it is possible this check to turn from a healthy green (OK) state to showing warnings after refreshing the VCG database. In such cases, VMware recommends to upgrade the firmware to the recommended version.

There is also the possibility of a vendor updating a firmware, and making it downloadable from their website. If VMware has not certified the firmware for vSAN (either because the certification has not happened yet, or because it actually failed the certification) VMware does not recommend upgrading to this firmware unless it appears on the VCG.

Refer to KB: https://kb.vmware.com/s/article/2149405

Environment

VMware Cloud Foundation 4.x

Cause

The upgrade process is requesting a function vSAN no longer provides with vSphere 7.x. Please see KB https://kb.vmware.com/s/article/2149405 for more information.

Resolution

Currently, there is no resolution.

Workaround:

1. Take a snapshot of the SDDC Manager
2. Login to SDDC Manager Login using vcf,  and su -
3. Change into the following directory:

cd /opt/vmware/vcf/lcm/lcm-app/conf/

4. Edit the file:

vi application-prod.properties

5. Search for the term: vsan
6. Change the values from true to false

Example changes:

vsan.healthcheck.enabled=false
vsan.hcl.update.enabled=false
vsan.precheck.enabled=false

7. Save the file
8. Restart the LCM service:

systemctl restart lcm

 

Manually check vSAN from vCenter to ensure no concerning issues exist.

Perform the upgrade from SDDC as normal. Since these steps are "skipped" there should be no more stops to the upgrade process for vSAN issues.

After completing the upgrade, return the values in the /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file to true again.

Restart the LCM service: systemctl restart lcm

 

Additional Information

Impact/Risks:
Customers have found upgrades impacted by this halt.

Best practice: manually check vSAN from vCenter before upgrade.