vSAN health service fails to start when upgrading to vCenter Server 6.7 U3

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter Server upgrade may fail due to vSAN health service cannot be started.
• In the /var/log/vmware/vsan-health/vmware-vsan-health-service.log, you see error message similar to:

2020-04-30T06:02:01.601Z ERROR vsan-mgmt[HclAutoUpdator] [VsanWpadUtil::getWpadProxy opID=noOpId] Fails in getting auto proxy for url:http://www.vmware.com, host:www.vmware.com with error:maximum recursion depth exceeded
Traceback (most recent call last):
File "/usr/lib/python3.5/site-packages/js2py/evaljs.py", line 173, in execute
compiled = cache[hashkey]
KeyError: b'\xbc;<\xaf\xeeI\xa9\xb8\xb8\xef_\xcd\xf9uh\xea'

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.7.x

Cause

This issue is caused by the third party python library js2py which is used for evaluating Java script in vSAN health service.
In this library, it may cause python stack overflow to crash the whole python server when evaluating large functions of java script.

Resolution

This is a known issue.

Currently, there is no resolution.

Workaround:
To workaround this issue:

1. SSH into vCenter Server.
2. Take a backup of config.conf file /etc/vmware-vsan-health/config.conf.
3. Edit  the config.conf file to add an invalid proxy IP into “[PhoneHome]” section as below:

[PhoneHome]
enablevsantelemetrycollectiondefault = False
vsantelemetrycollectionintervalinmindefault = 10080
proxyhost = 10.1.1.1      ----------------------------------------- Add an  invalid proxy IP
proxyport = 1000  

4. Save the file.
5. Restart vSAN health service by running the command:

 /usr/lib/vmware-vmon/vmon-cli -r vsan-health

6. Retry the Upgrade.