Failed to execute operation: Unit file is masked error when starting vmware-vmon.service

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After restoring VCSA 6.5 from backup, while starting the vmware-vmon.service service, you see entries on the logs similar to:
  
  YYYY-MM-DDTHH:MM:SS Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-vmon.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true']
  YYYY-MM-DDTHH:MM:SS Done running command
  YYYY-MM-DDTHH:MM:SS RC = 1
  Stdout =
  Stderr = Failed to execute operation: Unit file is masked
  YYYY-MM-DDTHH:MM:SS {
  "resolution": null,
  "detail": [
  {
  "args": [
  "Stderr: Failed to execute operation: Unit file is masked\n"
  ],
  "id": "install.ciscommon.command.errinvoke",
  "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'",
  "translatable": "An error occurred while invoking external command : '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
  YYYY-MM-DDTHH:MM:SS Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
  YYYY-MM-DDTHH:MM:SS Done running command
  Service-control failed. Error {
  "resolution": null,
  "detail": [
  {
  "args": [
  "vmware-vmon"
  ],
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'vmware-vmon'",
  "translatable": "An error occurred while starting service '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
   
• On the VCSA, the service status displays:
  
  Running:
  lwsmd vmafdd
  
  Stopped:
  applmgmt vmcam vmonapi vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

Resolution

To resolve the issue:

1. List the services that are masked by running the command:

​systemctl list-unit-files | grep masked

Output shows similar to:

applmgmt.service masked
vmcam.service masked
vmware-cm.service masked
vmware-content-library.service masked
vmware-eam.service masked
vmware-imagebuilder.service masked
vmware-mbcs.service masked
vmware-netdump.service masked
vmware-perfcharts.service masked
vmware-rbd-watchdog.service masked
vmware-rhttpproxy.service masked
vmware-sca.service masked
vmware-sps.service masked
vmware-statsmonitor.service masked
vmware-updatemgr.service masked
vmware-vapi.service masked
vmware-vcha.service masked
vmware-vmon.service masked
vmware-vmonapi.service masked
vmware-vpostgres.service masked
vmware-vpxd-svcs.service masked
vmware-vpxd.service masked
vmware-vsan-health.service masked
vmware-vsm.service masked
vsphere-client.service masked
vsphere-ui.service masked
ctrl-alt-del.target masked

2. Run these commands to list the services which are linked with /dev/null.​

cd /etc/systemd/system
/etc/systemd/system# ls -l

Output will be similar to:
total 84
180710 4 drwxr-xr-x 18 root root 4096 Aug 22 21:38 .
180226 4 drwxr-xr-x 7 root root 4096 Jun 6 15:56 ..
183074 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 applmgmt.service -> /dev/null
191121 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 applmgmt.service.d
182577 0 lrwxrwxrwx 1 root root 40 Feb 3 2017 default.target -> /usr/lib/systemd/system/runlevel3.target
180712 4 drwxr-xr-x 2 root root 4096 Jun 6 15:53 getty.target.wants
191133 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 halt.target.wants
188806 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 local-fs.target.wants
191117 4 drwxr-xr-x 2 root root 4096 Aug 22 22:21 lwsmd.service.d
180714 4 drwxr-xr-x 2 root root 4096 Jun 9 12:37 multi-user.target.wants
180718 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 network-online.target.wants
191135 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 poweroff.target.wants
191131 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 reboot.target.wants
182530 4 -rw-r--r-- 1 root root 268 Jun 7 2016 sendmail.service
191129 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 shutdown.target.wants
182807 4 -rw-r--r-- 1 root root 476 Aug 22 2016 snmpd.service
180720 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 sockets.target.wants
180722 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 sysinit.target.wants
182042 0 lrwxrwxrwx 1 root root 39 Feb 3 2017 syslog.service -> /usr/lib/systemd/system/rsyslog.service
182996 4 -r-xr-xr-x 1 root root 470 Apr 8 14:18 vcha-hacheck.service
320413 4 drwxr-xr-x 2 root root 4096 Aug 22 22:21 vmafdd.service.d
183095 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmcam.service -> /dev/null
183003 4 drwxr-xr-x 2 root root 4096 Jun 6 15:56 vmcam.service.d
182138 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 vmtoolsd.service.requires
183069 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-cm.service -> /dev/null
183091 0 lrwxrwxrwx 1 root root 9 Jun 6 17:01 vmware-content-library.service -> /dev/null
183098 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-eam.service -> /dev/null
183111 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-imagebuilder.service -> /dev/null
183080 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-mbcs.service -> /dev/null
183077 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-netdump.service -> /dev/null
183114 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-perfcharts.service -> /dev/null
183105 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-rbd-watchdog.service -> /dev/null
183068 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-rhttpproxy.service -> /dev/null
183070 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-sca.service -> /dev/null
182979 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-sps.service -> /dev/null
183078 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-statsmonitor.service -> /dev/null
183108 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-updatemgr.service -> /dev/null
183071 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-vapi.service -> /dev/null
183043 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-vcha.service -> /dev/null
183072 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vmonapi.service -> /dev/null
183130 0 lrwxrwxrwx 1 root root 9 Aug 22 21:38 vmware-vmon.service -> /dev/null
191125 4 drwxr-xr-x 2 root root 4096 Aug 22 22:37 vmware-vmon.service.d
183079 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vpostgres.service -> /dev/null
183090 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vmware-vpxd.service -> /dev/null
183029 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vpxd-svcs.service -> /dev/null
183109 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-vsan-health.service -> /dev/null
183110 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-vsm.service -> /dev/null
183087 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vsphere-client.service -> /dev/null
183089 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vsphere-ui.service -> /dev/null

3. ​Create a temp directory and move all the files that are linked to /dev/null to it. For example:

mkdir /temp
mv vsphere-ui.service /temp
mv vsphere-client.service /temp

      4. Restart the VCSA appliance.

Alternatively unmask the services with the below process identified from Step 1

     A.  Unmask the services with command

# find /etc/systemd/system/ -lname '/dev/null' -exec rm {} \; 

     B. Reload the process control agent

# systemctl daemon-reload

     C.  Stop and start all services

# service-control --stop --all && service-control --start --all