Vcenter is down after restore is failing with an error: An error occurred while starting service 'vmware-vmon'
search cancel

Vcenter is down after restore is failing with an error: An error occurred while starting service 'vmware-vmon'

book

Article ID: 381197

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Patching attempt was from version 8u3b to version 8u3d.
  • During the patching process, an error occurred:
    "Exception occurred in postInstallHook for B2B-patching. Please check the logs for more details. Take corrective action and then resume."
  • User attempted to roll back to the previous version, but the rollback timed out, resulting in the vCenter VAMI interface becoming unavailable.
  • A snapshot was created prior to patching to version 8u3d but when reverted back to it the services still failed to start.
  • Service status after the snapshot revert failure:
    service-control --status
    Running:
     lwsmd observability vmafdd vmdird
    Stopped:
     applmgmt lookupsvc observability-vapi pschealth vlcm vmcad vmcam vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-envoy vmware-envoy-hgw vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-pod vmware-postgres-archiver vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vdtc vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats vtsdb wcp
  • Error in PatchRunner.log captured:
    Installation failed. Retry to resume from the current state. Or please collect the VC support bundle.
 Mismatch: 
  summary: Internal error occurs during execution of update process Traceback (most recent call last):
  File "/storage/updatemgr/software-update35n8qnz7/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-update35n8qnz7/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 89, in _patchComponents
    _startDependentServices(c)
  File "/storage/updatemgr/software-update35n8qnz7/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 56, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/updatemgr/software-update35n8qnz7/stage/scripts/patches/libs/sdk/service_manager.py", line 909, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/updatemgr/software-update35n8qnz7/stage/scripts/patches/libs/sdk/service_manager.py", line 799, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/updatemgr/software-update35n8qnz7/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: 2022-08-24T07:03:17.854Z RC = 1
Stdout = 
Stderr = Job for vmware-vmon.service failed because the control process exited with error code.
See "systemctl status vmware-vmon.service" and "journalctl -xe" for details.
  • When checked in journalctl -xe:
    MMM DD XX:xx:xx <vcenter-name> vmon.launcher[61753]: /etc/profile.d/proxy.sh: eval: line 11: unexpected EOF while looking for matching `"'
MMM DD XX:xx:xx <vcenter-name> vmon.launcher[61753]: /etc/profile.d/proxy.sh: eval: line 12: syntax error: unexpected end of file
MMM DD XX:xx:xx <vcenter-name> systemd[1]: vmware-vmon.service: Control process exited, code=exited status=2
MMM DD XX:xx:xx <vcenter-name> systemd[1]: vmware-vmon.service: Failed with result 'exit-code'.

Environment

VMware vCenter Server 8.x

 

Cause

We identified a missing " in the proxy configuration as seen below in the proxy.sh file in user environment:

# Enable a generation of the proxy settings to the profile.
# This setting allows to turn the proxy on and off while
# preserving the particular proxy setup.
#
PROXY_ENABLED="yes"

# Some programs (e.g. wget) support proxies, if set in
# the environment.
# Example: HTTP_PROXY="http://proxy.provider.de:3128/"
HTTP_PROXY="http://xx.xxx.xxx.xx:xxxx"

# Example: HTTPS_PROXY="https://proxy.provider.de:3128/"
HTTPS_PROXY="https://xx.xxx.xxx.xx:xxxx ------------missing (") in the end of line----------------------

# Example: FTP_PROXY="http://proxy.provider.de:3128/"
FTP_PROXY=""

# Example: GOPHER_PROXY="http://proxy.provider.de:3128/"
GOPHER_PROXY=""

# Example: SOCKS_PROXY="socks://proxy.example.com:8080"
SOCKS_PROXY=""

# Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER=""

# Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost, 127.0.0.1"

 

Resolution

Edit /etc/sysconfig/proxy.sh file and add the missing " to the appropriate place as seen in the example below:

vi /etc/sysconfig/proxy.sh

Output:

Before the changes

# Enable a generation of the proxy settings to the profile.
# This setting allows to turn the proxy on and off while
# preserving the particular proxy setup.
#
PROXY_ENABLED="yes"

# Some programs (e.g. wget) support proxies, if set in
# the environment.
# Example: HTTP_PROXY="http://proxy.provider.de:3128/"
HTTP_PROXY="https://xx.xxx.xxx.xx:xxxx"

# Example: HTTPS_PROXY="https://proxy.provider.de:3128/"
HTTPS_PROXY="https://xx.xxx.xxx.xx:xxxx

# Example: FTP_PROXY="http://proxy.provider.de:3128/"
FTP_PROXY=""

# Example: GOPHER_PROXY="http://proxy.provider.de:3128/"
GOPHER_PROXY=""

# Example: SOCKS_PROXY="socks://proxy.example.com:8080"
SOCKS_PROXY=""

# Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER=""

# Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost, 127.0.0.1"

After the change:

# Enable a generation of the proxy settings to the profile.
# This setting allows to turn the proxy on and off while
# preserving the particular proxy setup.
#
PROXY_ENABLED="yes"

# Some programs (e.g. wget) support proxies, if set in
# the environment.
# Example: HTTP_PROXY="http://proxy.provider.de:3128/"
HTTP_PROXY="http://xx.xxx.xxx.xx:xxxx"

# Example: HTTPS_PROXY="https://proxy.provider.de:3128/"
HTTPS_PROXY="https://xx.xxx.xxx.xx:xxxx"

# Example: FTP_PROXY="http://proxy.provider.de:3128/"
FTP_PROXY=""

# Example: GOPHER_PROXY="http://proxy.provider.de:3128/"
GOPHER_PROXY=""

# Example: SOCKS_PROXY="socks://proxy.example.com:8080"
SOCKS_PROXY=""

# Example: SOCKS5_SERVER="office-proxy.example.com:8881"
SOCKS5_SERVER=""

# Example: NO_PROXY="www.me.de, do.main, localhost"
NO_PROXY="localhost, 127.0.0.1"
  • Post this restart all the services 
    service-control --stop --all && service-control --start --all
  • Retry the patching 
Powered by Wolken Software