"Failed to execute operation: Unit file is masked" or "ErrCode: 111" when we try to start the service on VCSA 6.X

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• When a vCenter Server is restored from backup the services will be stopped.
• When you try to start the services using the command : service-control --start --all you will see one of the following errors.
• Error 1:

YYYY-MM-DDTHH:MM:SS   Done running command
YYYY-MM-DDTHH:MM:SS   RC = 1
Stdout = 
Stderr = Failed to execute operation: Unit file is masked

2018-02-27T18:38:08.651Z   {
    "resolution": null, 
    "detail": [
        {
            "args": [
                "Stderr: Failed to execute operation: Unit file is masked\n"
            ], 
            "id": "install.ciscommon.command.errinvoke", 
            "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'", 
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        }
    ], 
    "componentKey": null, 
    "problemId": null
}
YYYY-MM-DDTHH:MM:SS   Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
YYYY-MM-DDTHH:MM:SS   Done running command
Service-control failed. Error {
    "resolution": null, 
    "detail": [
        {
            "args": [
                "vmware-vmon"
            ], 
            "id": "install.ciscommon.service.failstart", 
            "localized": "An error occurred while starting service 'vmware-vmon'", 
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ], 
    "componentKey": null, 
    "problemId": null
}
 

• Error 2

Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Service lwsmd startup type is not automatic. Skip
Performing start operation on service vmafdd...
Service vmafdd startup type is not automatic. Skip
Performing start operation on profile: ALL...
Service vmware-vmon startup type is not automatic. Skip
Service-control failed. Error: Failed to start services in profile ALL. RC=11, stderr=Connect error. ErrCode: 111

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x

Resolution

To resolve this issue,

1. List all disabled services for removal.   

find /etc/systemd/system/ -lname '/dev/null' -exec ls {} \;    

2. Automatically remove them (or rm each file).

find /etc/systemd/system/ -lname '/dev/null' -exec rm {} \;   

3. Reload systemctl daemon.

systemctl daemon-reload 

4. Start services or Reboot.

service-control --start --all  

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Impact/Risks:
Services wont start on vCenter Server.