vCenter services stooped working suddenly

Products

• VMware vCenter Server

Issue/Introduction

vCenter UI is not accessible with generic error.

Opening SSH session to check vCenter services with command "service-control --status --all"  you see that envoy services is not running.

Checking envoy logs located at /var/log/vmware/envoy you see similar entries 

 

critical envoy[#####] [Originator@6876 sub=backtrace] Caught Illegal instruction, suspect faulting address #x######
critical envoy[#####] [Originator@6876 sub=backtrace] Backtrace (use tools/stack decode.py to get line numbers):

 

After manually registering vCenter VM on different ESXI host vCenter boots up with no issues.

 

Checking host logs where vCenter VM was located at time of issue you see similar entries in vmkernel logs located at /var/run/log

 

Wa(180) vmkwarning: cpu54:2097271)WARNING: PageRetire: 624: Number of kernel MPNs selected for retirement is 1
Wa(180) vmkwarning: cpu54:2097271)WARNING: PageRetire: 624: Number of kernel MPNs selected for retirement is 2

 

For more information about vmkernel logs line please see 

VMkernel reports corrected memory errors and retirement of memory pages

Environment

vSphere 8.x

Cause

Envoy service was using faulty memory regions on ESXI host.

Resolution

Manually register vCenter VM on different host.