vmware-vmon, vmware-rhttpproxy and other services on vCenter fails to start

Products

• VMware vCenter Server

Issue/Introduction

• vCenter vmware-vmon service will crash immediately after the start, following this other services will fail to start as well.
• /var/log/vmware/vmon/vmon.log will not update with recent logs.
• vCenter service status will show as below :
  
  Running:
 lwsmd observability vmafdd vmcad vmdird vmware-vdtc
Stopped:
 applmgmt lookupsvc observability-vapi pschealth vlcm vmcam vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
  
  
• journalctl -xe log will show below information after vmware-vmon service crash:
  
  Mon DD HH:MM:SS vCenter_FQDN setupdb.sh[10500]: Creating VMON_TELEMETRY infrastructure.
Mon DD HH:MM:SS vCenter_FQDN setupdb.sh[10500]: DB Path is not set in arguments, using default path: /etc/vmware/vmware-vmon/telemet>
Mon DD HH:MM:SS vCenter_FQDN audit[10522]: SYSCALL arch=c000003e syscall=268 success=yes exit=0 a0=ffffff9c a1=5649f9b7c270 a2=1ed a>
Mon DD HH:MM:SS vCenter_FQDN audit: CWD cwd="/storage/log/vmware/vmon"
Mon DD HH:MM:SS vCenter_FQDN audit: PATH item=0 name="/var/run/vmware" inode=28039 dev=00:17 mode=040755 ouid=0 ogid=59001 rdev=00:0>
Mon DD HH:MM:SS vCenter_FQDN audit: PROCTITLE proctitle=63686D6F6400373535002F7661722F72756E2F766D77617265
Mon DD HH:MM:SS vCenter_FQDN audit[10523]: SYSCALL arch=c000003e syscall=260 success=yes exit=0 a0=ffffff9c a1=5566f4596d10 a2=fffff>
Mon DD HH:MM:SS vCenter_FQDN audit: CWD cwd="/storage/log/vmware/vmon"
Mon DD HH:MM:SS vCenter_FQDN audit: PATH item=0 name="/var/run/vmware" inode=28039 dev=00:17 mode=040755 ouid=0 ogid=59001 rdev=00:0>
Mon DD HH:MM:SS vCenter_FQDN audit: PROCTITLE proctitle=636867727000636973002F7661722F72756E2F766D77617265
Mon DD HH:MM:SS vCenter_FQDN vmon.launcher[10503]: rm: cannot remove '/storage/vmware-vmon/vmonserver': No such file or directory


• Listing the vCenter version or RPM details will throw following error:
  
  root@VC [ ~ ]# vpxd -vl
/usr/lib/vmware-vpx/vpxd: error while loading shared libraries: /lib/libzstd.so.1: file too short
root@VC [ ~ ]#

root@vc[ / ]# rpm -qa
rpm: error while loading shared libraries: /lib/libzstd.so.1: file too short
root@vc[ / ]#


• File libzstd.so.1.4.5 will show 0 in size:
  
  root@vc [ / ]# ls -ltrah /lib/libzstd.so.1
lrwxrwxrwx 1 root root 16 MM DD YYY /lib/libzstd.so.1 -> libzstd.so.1.4.5
root@vc [ / ]#
root@vc [ / ]# ls -ltrah /lib/libzstd.so.1.4.5
-rwxr-xr-x 1 root root 0 MM DD YYY /lib/libzstd.so.1.4.5
root@vcsa01 [ / ]#

Environment

VMware vCenter Server 7.x

Cause

This issue is caused due to a corrupt library libzstd.so.1.4.5 within the vCenter server appliance.

Resolution

Replace the library file "libzstd.so.1.4.5" from a working vCenter.

1. Use WinSCP or any SCP tool to connect to a vCenter server running the same build, then download /lib/libzstd.so.1.4.5 to your local machine.  
   In case of connection errors refer to : Toggling the vCenter Server Appliance default shell
   
   
2. Connect to the non-working vCenter using WinSCP and replace the file /lib/libzstd.so.1.4.5 with the previously copied file from the working vCenter.
   
   
3. Restart all the vCenter services
   
   service-control --stop --all
service-control --start --all