vMon Services Cannot Start and it fails with the error "Job for vmware-vmon.service failed because a configured resource limit was exceeded "
Article ID: 322818
Updated On:
Products
VMware vCenter Server
Issue/Introduction
This article help us in bringing up the vCenter services when the vmon services cannot start and it fails with the error message as described above.
Symptoms:
- The following services will be running lwsmd vmafdd vmcad vmdird vmdnsd vmware-sts-idmd vmware-stsd
- vMon service will fail with the below error during the startup of the services or when you try starting manually
Stderr = Job for vmware-vmon.service failed because a configured resource limit was exceeded. See "systemctl status vmware-vmon.service" and "journalctl -xe" for details.
2019-04-28T23:37:55.814Z {
"componentKey": null,
"resolution": null,
"detail": [
{
"localized": "An error occurred while invoking external command : 'Stderr: Job for vmware-vmon.service failed because a configured resource limit was exceeded. See \"systemctl status vmware-vmon.service\" and \"journalctl -xe\" for details.\n'",
"id": "install.ciscommon.command.errinvoke",
"translatable": "An error occurred while invoking external command : '%(0)s'",
"args": [
"Stderr: Job for vmware-vmon.service failed because a configured resource limit was exceeded. See \"systemctl status vmware-vmon.service\" and \"journalctl -xe\" for details.\n"
]
}
],
"problemId": null
2019-04-28T23:37:55.814Z {
"componentKey": null,
"resolution": null,
"detail": [
{
"localized": "An error occurred while invoking external command : 'Stderr: Job for vmware-vmon.service failed because a configured resource limit was exceeded. See \"systemctl status vmware-vmon.service\" and \"journalctl -xe\" for details.\n'",
"id": "install.ciscommon.command.errinvoke",
"translatable": "An error occurred while invoking external command : '%(0)s'",
"args": [
"Stderr: Job for vmware-vmon.service failed because a configured resource limit was exceeded. See \"systemctl status vmware-vmon.service\" and \"journalctl -xe\" for details.\n"
]
}
],
"problemId": null
- systemctl status vmware-vmon.service will show us the below error message
â vmware-vmon.service - VMware Service Lifecycle Manager
Loaded: loaded (/usr/lib/systemd/system/vmware-vmon.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/vmware-vmon.service.d
ââ50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dep_override.conf
Active: failed (Result: start-limit) since Mon 2019-04-29 05:20:03 UTC; 9s ago
Process: 49809 ExecStart=/usr/lib/vmware-vmon/vmon.launcher $VMON_PROFILE (code=exited, status=0/SUCCESS)
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: PID 49823 read from file /var/run/vmon.pid does not exist or is a zombie.
Apr 29 05:20:03 systemd[1]: Failed to start VMware Service Lifecycle Manager.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Unit entered failed state.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Failed with result 'resources'.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Service hold-off time over, scheduling restart.
Apr 29 05:20:03 systemd[1]: Stopped VMware Service Lifecycle Manager.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Start request repeated too quickly.
Apr 29 05:20:03 systemd[1]: Failed to start VMware Service Lifecycle Manager.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Unit entered failed state.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Failed with result 'start-limit'.
Loaded: loaded (/usr/lib/systemd/system/vmware-vmon.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/vmware-vmon.service.d
ââ50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf, dep_override.conf
Active: failed (Result: start-limit) since Mon 2019-04-29 05:20:03 UTC; 9s ago
Process: 49809 ExecStart=/usr/lib/vmware-vmon/vmon.launcher $VMON_PROFILE (code=exited, status=0/SUCCESS)
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: PID 49823 read from file /var/run/vmon.pid does not exist or is a zombie.
Apr 29 05:20:03 systemd[1]: Failed to start VMware Service Lifecycle Manager.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Unit entered failed state.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Failed with result 'resources'.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Service hold-off time over, scheduling restart.
Apr 29 05:20:03 systemd[1]: Stopped VMware Service Lifecycle Manager.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Start request repeated too quickly.
Apr 29 05:20:03 systemd[1]: Failed to start VMware Service Lifecycle Manager.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Unit entered failed state.
Apr 29 05:20:03 systemd[1]: vmware-vmon.service: Failed with result 'start-limit'.
- vmon.log shows below backtrace
2019-04-29T04:43:12.916964+00:00 notice vmon PANIC: VERIFY bora/vim/apps/vMon/src/ServiceManager.cpp:108
2019-04-29T04:43:12.917161+00:00 notice vmon Backtrace:
2019-04-29T04:43:12.917366+00:00 notice vmon Backtrace[0] 00007ffce9ddf3b0 rip=000056317f2acefe rbx=000056317f2acaf7 rbp=0000000000000000 r12=00007ffce9ddf818 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.917565+00:00 notice vmon Backtrace[1] 00007ffce9ddf3e0 rip=000056317f2aad1a rbx=00007ffce9ddf3e8 rbp=00007fd27cf02700 r12=00007ffce9ddf818 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.917760+00:00 notice vmon Backtrace[2] 00007ffce9ddf810 rip=000056317f2aac0a rbx=0000000000000019 rbp=000056318029f3e0 r12=00007ffce9ddfa01 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.917956+00:00 notice vmon Backtrace[3] 00007ffce9ddf8f0 rip=000056317f29e685 rbx=0000000000000019 rbp=000056318029f3e0 r12=00007ffce9ddfa01 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.918155+00:00 notice vmon Backtrace[4] 00007ffce9ddf9d0 rip=000056317f2a22b4 rbx=00007ffce9ddfa88 rbp=00007ffce9ddfa00 r12=00007ffce9ddfa08 r13=0000563180270100 r14=00007ffce9ddfa10 r15=00007ffce9ddfa28
2019-04-29T04:43:12.918362+00:00 notice vmon Backtrace[5] 00007ffce9ddfbe0 rip=000056317f294fb0 rbx=0000000000000000 rbp=000056317f2dcd10 r12=000056317f2955a8 r13=00007ffce9ddfcc0 r14=0000000000000000 r15=0000000000000000
2019-04-29T04:43:12.918558+00:00 notice vmon Backtrace[6] 00007ffce9ddfbf0 rip=00007fd27cb85570 rbx=0000000000000000 rbp=000056317f2dcd10 r12=000056317f2955a8 r13=00007ffce9ddfcc0 r14=0000000000000000 r15=0000000000000000
2019-04-29T04:43:12.918768+00:00 notice vmon Backtrace[7] 00007ffce9ddfcb0 rip=000056317f2955d1 rbx=0000000000000000 rbp=0000000000000000 r12=000056317f2955a8 r13=00007ffce9ddfcc0 r14=0000000000000000 r15=0000000000000000
2019-04-29T04:43:12.918973+00:00 notice vmon SymBacktrace[0] 00007ffce9ddf3b0 rip=000056317f2acefe in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919171+00:00 notice vmon SymBacktrace[1] 00007ffce9ddf3e0 rip=000056317f2aad1a in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919373+00:00 notice vmon SymBacktrace[2] 00007ffce9ddf810 rip=000056317f2aac0a in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919569+00:00 notice vmon SymBacktrace[3] 00007ffce9ddf8f0 rip=000056317f29e685 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919763+00:00 notice vmon SymBacktrace[4] 00007ffce9ddf9d0 rip=000056317f2a22b4 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.920017+00:00 notice vmon SymBacktrace[5] 00007ffce9ddfbe0 rip=000056317f294fb0 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.920226+00:00 notice vmon SymBacktrace[6] 00007ffce9ddfbf0 rip=00007fd27cb85570 in function __libc_start_main in object /lib/libc.so.6 loaded at 00007fd27cb65000
2019-04-29T04:43:12.920423+00:00 notice vmon SymBacktrace[7] 00007ffce9ddfcb0 rip=000056317f2955d1 in function (null) in object /usr/lib/vmware-:
2019-04-29T04:43:12.917161+00:00 notice vmon Backtrace:
2019-04-29T04:43:12.917366+00:00 notice vmon Backtrace[0] 00007ffce9ddf3b0 rip=000056317f2acefe rbx=000056317f2acaf7 rbp=0000000000000000 r12=00007ffce9ddf818 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.917565+00:00 notice vmon Backtrace[1] 00007ffce9ddf3e0 rip=000056317f2aad1a rbx=00007ffce9ddf3e8 rbp=00007fd27cf02700 r12=00007ffce9ddf818 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.917760+00:00 notice vmon Backtrace[2] 00007ffce9ddf810 rip=000056317f2aac0a rbx=0000000000000019 rbp=000056318029f3e0 r12=00007ffce9ddfa01 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.917956+00:00 notice vmon Backtrace[3] 00007ffce9ddf8f0 rip=000056317f29e685 rbx=0000000000000019 rbp=000056318029f3e0 r12=00007ffce9ddfa01 r13=000000000000001f r14=00007ffce9ddf978 r15=000056318029f3e8
2019-04-29T04:43:12.918155+00:00 notice vmon Backtrace[4] 00007ffce9ddf9d0 rip=000056317f2a22b4 rbx=00007ffce9ddfa88 rbp=00007ffce9ddfa00 r12=00007ffce9ddfa08 r13=0000563180270100 r14=00007ffce9ddfa10 r15=00007ffce9ddfa28
2019-04-29T04:43:12.918362+00:00 notice vmon Backtrace[5] 00007ffce9ddfbe0 rip=000056317f294fb0 rbx=0000000000000000 rbp=000056317f2dcd10 r12=000056317f2955a8 r13=00007ffce9ddfcc0 r14=0000000000000000 r15=0000000000000000
2019-04-29T04:43:12.918558+00:00 notice vmon Backtrace[6] 00007ffce9ddfbf0 rip=00007fd27cb85570 rbx=0000000000000000 rbp=000056317f2dcd10 r12=000056317f2955a8 r13=00007ffce9ddfcc0 r14=0000000000000000 r15=0000000000000000
2019-04-29T04:43:12.918768+00:00 notice vmon Backtrace[7] 00007ffce9ddfcb0 rip=000056317f2955d1 rbx=0000000000000000 rbp=0000000000000000 r12=000056317f2955a8 r13=00007ffce9ddfcc0 r14=0000000000000000 r15=0000000000000000
2019-04-29T04:43:12.918973+00:00 notice vmon SymBacktrace[0] 00007ffce9ddf3b0 rip=000056317f2acefe in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919171+00:00 notice vmon SymBacktrace[1] 00007ffce9ddf3e0 rip=000056317f2aad1a in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919373+00:00 notice vmon SymBacktrace[2] 00007ffce9ddf810 rip=000056317f2aac0a in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919569+00:00 notice vmon SymBacktrace[3] 00007ffce9ddf8f0 rip=000056317f29e685 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.919763+00:00 notice vmon SymBacktrace[4] 00007ffce9ddf9d0 rip=000056317f2a22b4 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.920017+00:00 notice vmon SymBacktrace[5] 00007ffce9ddfbe0 rip=000056317f294fb0 in function (null) in object /usr/lib/vmware-vmon/vmon loaded at 000056317f280000
2019-04-29T04:43:12.920226+00:00 notice vmon SymBacktrace[6] 00007ffce9ddfbf0 rip=00007fd27cb85570 in function __libc_start_main in object /lib/libc.so.6 loaded at 00007fd27cb65000
2019-04-29T04:43:12.920423+00:00 notice vmon SymBacktrace[7] 00007ffce9ddfcb0 rip=000056317f2955d1 in function (null) in object /usr/lib/vmware-:
Environment
vCenter Server 6.x
vCenter Server 7.x
vCenter Server 8.x
Cause
- This occurs when the files under /etc/vmware/vmware-vmon/svcCfgfiles/ have been modified and the backup file for any one of the json file is present
- There was backup file statsmonitor.json.orig under /etc/vmware/vmware-vmon/svcCfgfiles/
- All the below files should be present and there should not be any files other than below and it should look exactly the same
-r--r--r-- 1 root root 3438 Mar 15 2018 analytics.json
-r--r--r-- 1 root root 975 Apr 9 2018 applmgmt.json
-rw-r--r-- 1 root root 780 Apr 3 2018 autodeploy.json
-r--r--r-- 1 root root 3030 Mar 15 2018 cis-license.json
-r--r--r-- 1 cm cis 4263 Apr 9 2018 cm.json
-rw-r--r-- 1 eam cis 3637 Apr 9 2018 eam.json
-r--r--r-- 1 root root 913 Apr 9 2018 imagebuilder.json
-rw-r--r-- 1 mbcs cis 4089 Apr 9 2018 mbcs.json
-r--r--r-- 1 root root 859 Apr 9 2018 netdumper.json
-rwxr-xr-x 1 root root 4875 Apr 9 2018 perfcharts.json
-rw-r--r-- 1 root root 681 Mar 30 2018 pschealth.json
-r--r--r-- 1 root root 1530 Apr 9 2018 rhttpproxy.json
-rwxr-xr-x 1 root root 5573 Apr 9 2018 sps.json
-r--r--r-- 1 root root 1024 Apr 9 2018 statsmonitor.json
-rw-r--r-- 1 root root 834 Apr 3 2018 updatemgr.json
-r--r--r-- 1 root root 7122 Apr 9 2018 vapi-endpoint.json
-r--r--r-- 1 root root 837 Apr 9 2018 vcha.json
-r--r----- 1 content-library cis 4467 Apr 9 2018 vdcs.json
-rw-r--r-- 1 root root 674 Feb 27 2018 vmcam.json
-rw-r--r-- 1 root root 1071 Apr 9 2018 vmonapi.json
-rw-r--r-- 1 root root 1738 Jan 29 2018 vmware-postgres-archiver.json
-r--r--r-- 1 root root 3000 Dec 11 2017 vmware-sca.json
-rw-r--r-- 1 root root 1169 Jan 29 2018 vmware-vpostgres.json
-r--r--r-- 1 root root 1057 Apr 9 2018 vpxd.json
-rw-r--r-- 1 root root 4174 Apr 9 2018 vpxd-svcs.json
-r--r----- 1 vsan-dps cis 3942 Apr 9 2018 vsan-dps.json
-r--r--r-- 1 root root 798 Apr 9 2018 vsan-health.json
-r--r--r-- 1 vsm cis 3649 Apr 9 2018 vsm.json
-rw-r--r-- 1 root root 6129 Apr 3 2018 vsphere-client.json
-rw-r--r-- 1 root root 5910 Apr 3 2018 vsphere-ui.json
-r--r--r-- 1 root root 975 Apr 9 2018 applmgmt.json
-rw-r--r-- 1 root root 780 Apr 3 2018 autodeploy.json
-r--r--r-- 1 root root 3030 Mar 15 2018 cis-license.json
-r--r--r-- 1 cm cis 4263 Apr 9 2018 cm.json
-rw-r--r-- 1 eam cis 3637 Apr 9 2018 eam.json
-r--r--r-- 1 root root 913 Apr 9 2018 imagebuilder.json
-rw-r--r-- 1 mbcs cis 4089 Apr 9 2018 mbcs.json
-r--r--r-- 1 root root 859 Apr 9 2018 netdumper.json
-rwxr-xr-x 1 root root 4875 Apr 9 2018 perfcharts.json
-rw-r--r-- 1 root root 681 Mar 30 2018 pschealth.json
-r--r--r-- 1 root root 1530 Apr 9 2018 rhttpproxy.json
-rwxr-xr-x 1 root root 5573 Apr 9 2018 sps.json
-r--r--r-- 1 root root 1024 Apr 9 2018 statsmonitor.json
-rw-r--r-- 1 root root 834 Apr 3 2018 updatemgr.json
-r--r--r-- 1 root root 7122 Apr 9 2018 vapi-endpoint.json
-r--r--r-- 1 root root 837 Apr 9 2018 vcha.json
-r--r----- 1 content-library cis 4467 Apr 9 2018 vdcs.json
-rw-r--r-- 1 root root 674 Feb 27 2018 vmcam.json
-rw-r--r-- 1 root root 1071 Apr 9 2018 vmonapi.json
-rw-r--r-- 1 root root 1738 Jan 29 2018 vmware-postgres-archiver.json
-r--r--r-- 1 root root 3000 Dec 11 2017 vmware-sca.json
-rw-r--r-- 1 root root 1169 Jan 29 2018 vmware-vpostgres.json
-r--r--r-- 1 root root 1057 Apr 9 2018 vpxd.json
-rw-r--r-- 1 root root 4174 Apr 9 2018 vpxd-svcs.json
-r--r----- 1 vsan-dps cis 3942 Apr 9 2018 vsan-dps.json
-r--r--r-- 1 root root 798 Apr 9 2018 vsan-health.json
-r--r--r-- 1 vsm cis 3649 Apr 9 2018 vsm.json
-rw-r--r-- 1 root root 6129 Apr 3 2018 vsphere-client.json
-rw-r--r-- 1 root root 5910 Apr 3 2018 vsphere-ui.json
- We had one more in there named as statsmonitor.json.orig in-addition to statsmonitor.json
For Example
-r--r--r-- 1 smotethinaana mts 1070 May 15 19:34 vpxd.json
-r--r--r-- 1 smotethinaana mts 1046 May 16 05:48 vpxd-modified.json ---> This file is causing the problem.
-rw-r--r-- 1 smotethinaana mts 4191 May 10 22:17 vpxd-svcs.json
-r--r--r-- 1 smotethinaana mts 1070 May 15 19:34 vpxd.json
-r--r--r-- 1 smotethinaana mts 1046 May 16 05:48 vpxd-modified.json ---> This file is causing the problem.
-rw-r--r-- 1 smotethinaana mts 4191 May 10 22:17 vpxd-svcs.json
Resolution
- Move any additional files (vpxd-modified.json or statsmonitor.json.ori)which are present under /etc/vmware/vmware-vmon/svcCfgfiles/ which are created as backup files to /storage/log
- Start all the vCenter Services again
service-control --start --all
Additional Information
Impact/Risks:
None of the original json files should be modified or deleted.
None of the original json files should be modified or deleted.
Feedback
Yes
No
Powered by
