Failed to execute operation: Unit file is masked error when starting vmware-vmon.service

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After restoring VCSA 6.5 from backup, while starting the vmware-vmon.service service, you see entries on the logs similar to:
  
  2017-08-22T22:21:28.912Z Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-vmon.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true']
  2017-08-22T22:21:28.916Z Done running command
  2017-08-22T22:21:28.918Z RC = 1
  Stdout =
  Stderr = Failed to execute operation: Unit file is masked
  2017-08-22T22:21:28.919Z {
  "resolution": null,
  "detail": [
  {
  "args": [
  "Stderr: Failed to execute operation: Unit file is masked\n"
  ],
  "id": "install.ciscommon.command.errinvoke",
  "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'",
  "translatable": "An error occurred while invoking external command : '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
  2017-08-22T22:21:28.919Z Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
  2017-08-22T22:21:28.921Z Done running command
  Service-control failed. Error {
  "resolution": null,
  "detail": [
  {
  "args": [
  "vmware-vmon"
  ],
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'vmware-vmon'",
  "translatable": "An error occurred while starting service '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
   
• On the VCSA, the service status displays:
  
  Running:
  lwsmd vmafdd
  
  Stopped:
  applmgmt vmcam vmonapi vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.5.x
vCenter 7.x

Resolution

To resolve the issue:

1. List the services that are masked by running the command:

​systemctl list-unit-files | grep masked

Output shows similar to:

applmgmt.service masked
vmcam.service masked
vmware-cm.service masked
vmware-content-library.service masked
vmware-eam.service masked
vmware-imagebuilder.service masked
vmware-mbcs.service masked
vmware-netdump.service masked
vmware-perfcharts.service masked
vmware-rbd-watchdog.service masked
vmware-rhttpproxy.service masked
vmware-sca.service masked
vmware-sps.service masked
vmware-statsmonitor.service masked
vmware-updatemgr.service masked
vmware-vapi.service masked
vmware-vcha.service masked
vmware-vmon.service masked
vmware-vmonapi.service masked
vmware-vpostgres.service masked
vmware-vpxd-svcs.service masked
vmware-vpxd.service masked
vmware-vsan-health.service masked
vmware-vsm.service masked
vsphere-client.service masked
vsphere-ui.service masked
ctrl-alt-del.target masked

2. Run these commands to list the services which are linked with /dev/null.​

cd /etc/systemd/system
/etc/systemd/system# ls -l

Output will be similar to:
total 84
180710 4 drwxr-xr-x 18 root root 4096 Aug 22 21:38 .
180226 4 drwxr-xr-x 7 root root 4096 Jun 6 15:56 ..
183074 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 applmgmt.service -> /dev/null
191121 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 applmgmt.service.d
182577 0 lrwxrwxrwx 1 root root 40 Feb 3 2017 default.target -> /usr/lib/systemd/system/runlevel3.target
180712 4 drwxr-xr-x 2 root root 4096 Jun 6 15:53 getty.target.wants
191133 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 halt.target.wants
188806 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 local-fs.target.wants
191117 4 drwxr-xr-x 2 root root 4096 Aug 22 22:21 lwsmd.service.d
180714 4 drwxr-xr-x 2 root root 4096 Jun 9 12:37 multi-user.target.wants
180718 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 network-online.target.wants
191135 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 poweroff.target.wants
191131 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 reboot.target.wants
182530 4 -rw-r--r-- 1 root root 268 Jun 7 2016 sendmail.service
191129 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 shutdown.target.wants
182807 4 -rw-r--r-- 1 root root 476 Aug 22 2016 snmpd.service
180720 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 sockets.target.wants
180722 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 sysinit.target.wants
182042 0 lrwxrwxrwx 1 root root 39 Feb 3 2017 syslog.service -> /usr/lib/systemd/system/rsyslog.service
182996 4 -r-xr-xr-x 1 root root 470 Apr 8 14:18 vcha-hacheck.service
320413 4 drwxr-xr-x 2 root root 4096 Aug 22 22:21 vmafdd.service.d
183095 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmcam.service -> /dev/null
183003 4 drwxr-xr-x 2 root root 4096 Jun 6 15:56 vmcam.service.d
182138 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 vmtoolsd.service.requires
183069 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-cm.service -> /dev/null
183091 0 lrwxrwxrwx 1 root root 9 Jun 6 17:01 vmware-content-library.service -> /dev/null
183098 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-eam.service -> /dev/null
183111 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-imagebuilder.service -> /dev/null
183080 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-mbcs.service -> /dev/null
183077 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-netdump.service -> /dev/null
183114 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-perfcharts.service -> /dev/null
183105 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-rbd-watchdog.service -> /dev/null
183068 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-rhttpproxy.service -> /dev/null
183070 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-sca.service -> /dev/null
182979 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-sps.service -> /dev/null
183078 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-statsmonitor.service -> /dev/null
183108 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-updatemgr.service -> /dev/null
183071 0 lrwxrwxrwx 1 root root 9 Jun 6 16:58 vmware-vapi.service -> /dev/null
183043 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-vcha.service -> /dev/null
183072 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vmonapi.service -> /dev/null
183130 0 lrwxrwxrwx 1 root root 9 Aug 22 21:38 vmware-vmon.service -> /dev/null
191125 4 drwxr-xr-x 2 root root 4096 Aug 22 22:37 vmware-vmon.service.d
183079 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vpostgres.service -> /dev/null
183090 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vmware-vpxd.service -> /dev/null
183029 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vpxd-svcs.service -> /dev/null
183109 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-vsan-health.service -> /dev/null
183110 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-vsm.service -> /dev/null
183087 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vsphere-client.service -> /dev/null
183089 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vsphere-ui.service -> /dev/null

3. ​Create a temp directory and move all the files that are linked to /dev/null to it. For example:

mkdir /temp
mv vsphere-ui.service /temp
mv vsphere-client.service /temp

      4. Restart the VCSA appliance.

Alternatively unmask the services with the below process identified from Step 1

     A.  Unmask the services with command

# find /etc/systemd/system/ -lname '/dev/null' -exec rm {} \; 

     B. Reload the process control agent

# systemctl daemon-reload

     C.  Stop and start all services

# service-control --stop --all && service-control --start --all