Unable to perform upgrade due to vSAN issue

Products

• VMware vSAN
• VMware vCenter Server

Issue/Introduction

Customer is experiencing an error in the vCenter Web Client when trying to upgrade vCenter from 7.X to 8.X.

Reason is: com.vmware.vcf.error.environment.vsan.hcl.update.failure

Environment

vCenter 7.X

ESXi 7.X

Cause

In vCenter, there were hundreds of .gz logs in the  /var/log/vmware/vsan-health directory.

EX:
vsanvcmgmtd-183.log.gz
vsanvcmgmtd-184.log.gz
vsanvcmgmtd-185.log.gz

The compressed .gz files located at /var/log/vmware/vsan-health on the vCenter Server can grow rapidly and cause various symptoms such as upgrade issues, cosmetic issues in vCenter such as false alerts, vCenter becoming inaccessible or the vSAN health service failing to start or restart.  

The connection between this directory having so many compressed files and upgrade issues is still under investigation.

Resolution

Move .gz logs from vCenter directory /var/log/vmware/vsan-health to an external storage device or if not needed, they can be deleted.

After moving or deleting the .gz files from the vCenter directory /var/log/vmware/vsan-health, the vCenter upgrade is able to complete successfully.