Error: "Failed to execute operation: Unit file is masked" when starting vmware-vmon.service

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After restoring VCSA 6.x from backup, while starting the vmware-vmon.service service, there may be entries in the logs similar to:
  
  [YYYY-MM-DDTHH:MM:SS] Running command: ['/usr/bin/systemctl', 'set-property', u'vmware-vmon.service', 'MemoryAccounting=true', 'CPUAccounting=true', 'BlockIOAccounting=true']
  [YYYY-MM-DDTHH:MM:SS] RC = 1
  Stdout =
  Stderr = Failed to execute operation: Unit file is masked
  YYYY-MM-DDTHH:MM:SS {
  "resolution": null,
  "detail": [
  {
  "args": [
  "Stderr: Failed to execute operation: Unit file is masked\n"
  ],
  "id": "install.ciscommon.command.errinvoke",
  "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'",
  "translatable": "An error occurred while invoking external command : '%(0)s'"
  }
  ],
  
  }
  [YYYY-MM-DDTHH:MM:SS] Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
  [YYYY-MM-DDTHH:MM:SS] Done running command
  Service-control failed. Error {
  "resolution": null,
  "detail": [
  {
  "args": [
  "vmware-vmon"
  ],
  "id": "install.ciscommon.service.failstart",
  "localized": "An error occurred while starting service 'vmware-vmon'",
  "translatable": "An error occurred while starting service '%(0)s'"
  }
  ],
  "componentKey": null,
  "problemId": null
  }
   
• On the vCenter server, the service-control --status command may display:
  
  Running:
lwsmd vmafdd

Stopped:
applmgmt vmcam vmonapi vmware-cm vmware-content-library vmware-eam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x

Cause

System files may become masked and not able to start.

Resolution

To resolve the issue:

1. List the services that are masked:

​# systemctl list-unit-files | grep masked

 

Output will be similar to:

applmgmt.service masked
vmcam.service masked
vmware-cm.service masked
vmware-content-library.service masked
...

vsphere-client.service masked
vsphere-ui.service masked
ctrl-alt-del.target masked

2. List the services which are linked to /dev/null:

         # ls -l /etc/systemd/system

Output will be similar to:
total 84
180710 4 drwxr-xr-x 18 root root 4096 Aug 22 21:38 .
180226 4 drwxr-xr-x 7 root root 4096 Jun 6 15:56 ..
183074 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 applmgmt.service -> /dev/null
191121 4 drwxr-xr-x 2 root root 4096 Jun 6 17:04 applmgmt.service.d
182577 0 lrwxrwxrwx 1 root root 40 Feb 3 2017 default.target -> /usr/lib/systemd/system/runlevel3.target
...
182807 4 -rw-r--r-- 1 root root 476 Aug 22 2016 snmpd.service
180720 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 sockets.target.wants
180722 4 drwxr-xr-x 2 root root 4096 Feb 3 2017 sysinit.target.wants
182042 0 lrwxrwxrwx 1 root root 39 Feb 3 2017 syslog.service -> /usr/lib/systemd/system/rsyslog.service
182996 4 -r-xr-xr-x 1 root root 470 Apr 8 14:18 vcha-hacheck.service
320413 4 drwxr-xr-x 2 root root 4096 Aug 22 22:21 vmafdd.service.d
183095 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmcam.service -> /dev/null
183111 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-imagebuilder.service -> /dev/null
183080 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-mbcs.service -> /dev/null
...
183029 0 lrwxrwxrwx 1 root root 9 Jun 6 16:59 vmware-vpxd-svcs.service -> /dev/null
183109 0 lrwxrwxrwx 1 root root 9 Jun 6 17:02 vmware-vsan-health.service -> /dev/null
183110 0 lrwxrwxrwx 1 root root 9 Jun 6 17:03 vmware-vsm.service -> /dev/null
183087 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vsphere-client.service -> /dev/null
183089 0 lrwxrwxrwx 1 root root 9 Jun 6 17:00 vsphere-ui.service -> /dev/null

3. ​Create a temp directory and move all the files that are linked to /dev/null :

# mkdir /temp
# mv vsphere-ui.service /temp
# mv vsphere-client.service /temp

      4. Restart the vCenter server.