Copies of vmon service configuration files in /etc/vmware/vmware-vmon/svcCfgfiles/ causes vCenter services to fail to start

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Services fail to start after vmon is initiated with service-control

Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Read error. ErrCode: 104

 

• The /var/log/vmware/vmon/vmon.log file contains errors similar to the following

2022-04-13T18:58:56.111Z In(05) host-XXXX Adding service sps.
2022-04-13T18:58:56.111Z In(05) host-XXXX PANIC: VERIFY bora/vim/apps/vMon/src/ServiceManager.cpp:110
2022-04-13T18:58:56.111Z In(05) host-XXXX Backtrace:
2022-04-13T18:58:56.111Z In(05) host-XXXX Backtrace[0] 00007ffc99bd10c0 rip=000055b94d6bef42 rbx=000000000000002a rbp=00000000000000c8 r12=00007000000060 r13=0000000000000f8 r14=0000000000000021 r15=000000000000098
...
2022-04-13T18:58:56.111Z In(05) host-XXXX SymBacktrace[6] 00007ffc99bd1940 rip=000055b94d6696f1 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000000000000000
2022-04-13T18:58:56.111Z Wa(03) host-XXXX
2022-04-13T18:58:56.111Z Wa(03)+ host-XXXX Attempting a core dump using an unlimited size limit
2022-04-13T18:58:57.112Z Wa(03) host-XXXX Core dumped.
2022-04-13T18:58:57.112Z Wa(03) host-XXXX A core file is available in "/var/core/core.vmon.12345"
2022-04-13T18:58:57.112Z In(05) host-XXXX Msg_Post: Error
2022-04-13T18:58:57.112Z In(05) host-XXXX [msg.log.error.unrecoverable] VMware Workstation unrecoverable error: (host-XXXX)
2022-04-13T18:58:57.112Z In(05)+ host-XXXX VERIFY bora/vim/apps/vMon/src/ServiceManager.cpp:110
2022-04-13T18:58:57.112Z In(05) host-XXXX [msg.panic.haveLog] A log file is available in "vmon.log".
2022-04-13T18:58:57.112Z In(05) host-XXXX [msg.panic.requestSupport.withoutLog] You can request support.
2022-04-13T18:58:57.112Z In(05) host-XXXX [msg.panic.requestSupport.vmSupport.windowsOrLinux]
 

Environment

VMware vCenter Server 7.0.0

Cause

When vmon starts, it reads the files in /etc/vmware/vmware-vmon/svcCfgfiles/ and tries to load any that contain service configurations. If there are 2 files for the same service, vmon will panic and fail to start. This is likely due to someone creating a backup of the file before modifying the original.

Resolution

To resolve this issue, remove or move the erroneous files to a new location such as /root or /tmp.

Additional Information

• A random file in this directory should not cause a problem unless it's identified by vmon as usable.
• Messages like the following pertaining to the jmx directory can safely be ignored.

2022-04-13T19:05:44.717Z Er(02) host-XXXX Error reading file. Details Is a directory
2022-04-13T19:05:44.717Z Er(02) host-XXXX Failed to read json config /etc/vmware/vmware-vmon/svcCfgfiles/jmx
2022-04-13T19:05:44.717Z Er(02) host-XXXX Invalid service config file: jmx
2022-04-13T19:05:44.717Z Wa(03) host-XXXX Failed to add service jmx.