"Failed to execute operation: Unit file is masked" or "ErrCode: 111" when trying to start the service on VCSA 6.X

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 6.0

Issue/Introduction

 

• When vCenter Server is restored from backup the services will be stopped.
• When you try to start the services using the command: service-control --start --all you will see one of the following errors.

[YYYY-MM-DDTHH:MM:SS]   Done running command
[YYYY-MM-DDTHH:MM:SS]   RC = 1
Stderr = Failed to execute operation: Unit file is masked

[YYYY-MM-DDTHH:MM:SS]  {
    "resolution": null, 
    "detail": [
        {
            "args": [
                "Stderr: Failed to execute operation: Unit file is masked\n"
            ], 
            "id": "install.ciscommon.command.errinvoke", 
            "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'", 
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        }
    ], 

}
[YYYY-MM-DDTHH:MM:SS]   Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
[YYYY-MM-DDTHH:MM:SS]   Done running command
Service-control failed. Error {
    "resolution": null, 
    "detail": [
        {
            "args": [
                "vmware-vmon"
            ], 
            "id": "install.ciscommon.service.failstart", 
            "localized": "An error occurred while starting service 'vmware-vmon'", 
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ], 
    "componentKey": null, 
    "problemId": null
}
 Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Service lwsmd startup type is not automatic. Skip
Performing start operation on service vmafdd...
Service vmafdd startup type is not automatic. Skip
Performing start operation on profile: ALL...
Service vmware-vmon startup type is not automatic. Skip
Service-control failed. Error: Failed to start services in profile ALL. RC=11, stderr=Connect error. ErrCode: 111

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x

Cause

Services become masked and unable to start.

Resolution

To resolve this issue:

1. cd to /etc/systemd/system
2. # systemctl list-unit-files | grep masked
3. # find /etc/systemd/system/ -lname '/dev/null' -exec ls {} \;
4. # find /etc/systemd/system/ -lname '/dev/null' -exec rm {} \;
5. # systemctl daemon-reload
6. # service-control --stop --all && service-control --start --all

Additional Information

Commands:

List masked services:
[ /etc/systemd/system ]# systemctl list-unit-files | grep masked

Remove masked services:
[ /etc/systemd/system ]# find /etc/systemd/system/ -lname '/dev/null' -exec ls {} \;