vSAN service logs on vCenter grow rapidly to large size and do not rotate as expected.

Products

• VMware vSAN

Issue/Introduction

This KB is written to provide notice that this issue is known, provide a workaround, and information on where the issue is resolved.

Symptoms:
The following service logs grow rapidly and fill the /storage/log directory on vCenter resulting in vCenter and vCenter services crash and services being unable to start or restart.
Log files: vmware-vsan-health-runtimelog, vmware-vsan-health-service.log , vmware-vsan-health-summary-result.log, and vsanvcmgmtd.log

You will see a lack of recent compressed log files in the .gz or .tgz format created.
You will see an extremely large log file such as 10+ GB 

Example (your file will have different size and date stamp, possible different vSAN file name):
In /storage/log/vmware/vsan-health/ directory:
14663550211 Dec 14 19:19 vmware-vsan-health-service.log

Environment

VMware vSAN 6.7.x
VMware vSAN 7.0.x

Cause

In certain circumstances the /etc/vmware-vsan-health/logger.conf configuration file is not updated to the new format used by the vSAN health service on vCenter after an update. This leads to situations where the log rotation does not work as expected.

Resolution

This issue is resolved in vCenter server version 6.7U3j and later, and 7.0 U1 and later where even if an older configuration file is present, it will be updated to the new format.

Workaround:
As a workaround the following process may be followed:

1. Remove all the huge log files under /var/log/vmware/vsan-health
2. Remove /etc/vmware-vsan-health/logger.conf
3. Restart vsan-health with: vmon-cli -r vsan-health

Additional Information

Impact/Risks:
This issue can cause vCenter services to crash and be unable to restart.