vCenter HA failover happened without reason.
From the vmon log in previous active vcenter, you can see below log entry
Error: Failed to read health xml file: /etc/vmware-vcha/vcha_health.xml. Error: time data '[DATE]T23:54:35Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
- /var/log/vmware/vmon.log
[DATE/TIME] In(05) host-16370 <vcha-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-vcha -f /etc/vmware-vcha/vcha_health.xml
[DATE/TIME] Wa(03) host-16370 <vcha> Service api-health command's stderr: Error getting service health. Error: Failed to read health xml file: /etc/vmware-vcha/vcha_health.xml. Error: time data '[DATE]T23:54:35Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
[DATE/TIME] Wa(03)+ host-16370
[DATE/TIME] Wa(03) host-16370 <vcha> Service api healthcheck command returned unknown exit code 1
[DATE/TIME] Er(02) host-16370 <vcha> health state unknown ,considered as system failure
[DATE/TIME] Er(02) host-16370 System Failure, initiating system restart.
VMware vCenter Server 8.0
VMware vCenter Server 9.0
As you can see below log, the time value does not have msec and usec value.
Error: Failed to read health xml file: /etc/vmware-vcha/vcha_health.xml. Error: time data '[DATE]T23:54:35Z' does not match format '%Y-%m-%dT%H:%M:%S.%fZ'
It is violation for parsing date and the possibility of this issue can be only one value from our code side.
The value is like '[DATE]T23:54:35.000.000Z'
There is no workaround and this can happen at a very rare probability.
This issue is resolved in releases below:
vCenter Server 8.0 Update 3h
VMware vCenter 9.0.1.0