vCenter Server services fail to start with "Unit file is masked" or "ErrCode: 111"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 6.0
• VMware vCenter Server 8.0

Issue/Introduction

• vCenter Server services may fail to start during normal operations or following a backup restoration, resulting in an inaccessible management interface (VAMI) and vCenter UI.
  
  
• When trying to start the services using the command: service-control --start --all the following errors are seen: 

[YYYY-MM-DDTHH:MM:SS]   Done running command
[YYYY-MM-DDTHH:MM:SS]   RC = 1
Stderr = Failed to execute operation: Unit file is masked

[YYYY-MM-DDTHH:MM:SS]  {
    "resolution": null, 
    "detail": [
        {
            "args": [
                "Stderr: Failed to execute operation: Unit file is masked\n"
            ], 
            "id": "install.ciscommon.command.errinvoke", 
            "localized": "An error occurred while invoking external command : 'Stderr: Failed to execute operation: Unit file is masked\n'", 
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        }
    ], 

}
[YYYY-MM-DDTHH:MM:SS]   Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
[YYYY-MM-DDTHH:MM:SS]   Done running command
Service-control failed. Error {
    "resolution": null, 
    "detail": [
        {
            "args": [
                "vmware-vmon"
            ], 
            "id": "install.ciscommon.service.failstart", 
            "localized": "An error occurred while starting service 'vmware-vmon'", 
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ], 
    "componentKey": null, 
    "problemId": null
}
 Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Service lwsmd startup type is not automatic. Skip
Performing start operation on service vmafdd...
Service vmafdd startup type is not automatic. Skip
Performing start operation on profile: ALL...
Service vmware-vmon startup type is not automatic. Skip
Service-control failed. Error: Failed to start services in profile ALL. RC=11, stderr=Connect error.
ErrCode: 111

• Running service-control --status --all shows lookupsvc stuck in Start Pending status, while other critical services like vmafdd are Stopped.
  
  # service-control --status --all
Running:
 observability vmware-envoy vmware-envoy-hgw vmware-envoy-sidecar vmware-postgres-archiver vmware-rhttpproxy vmware-vdtc vmware-vmon vmware-vpostgres vtsdb
StartPending:
 lookupsvc
Stopped:
 applmgmt lwsmd observability-vapi pschealth vc-ws1a-broker vlcm vmafdd vmcad vmcam vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-content-library vmware-eam vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-sca vmware-sps vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats wcp


• Reviewing /var/log/vmware/lookupsvc/lookupsvc.log reveals connectivity failures to the Authentication Framework Daemon
  
  [YYYY-MM-DDTHH:MM:SS] WARNING  Endpoint /lookupservice/sdk responded unexpectedly to GET with http status 404
[YYYY-MM-DDTHH:MM:SS] ERROR    Failed to execute command '['/usr/lib/vmware-vmafd/bin/vmafd-cli', 'get-dc-name', '--server-name', 'localhost']'
[YYYY-MM-DDTHH:MM:SS] ERROR
[YYYY-MM-DDTHH:MM:SS] ERROR    Error 1021: Could not connect to the local service VMware AFD.
Verify VMware AFD is running.
[YYYY-MM-DDTHH:MM:SS] ERROR    [['/usr/lib/vmware-vmafd/bin/vmafd-cli', 'get-dc-name', '--server-name', 'localhost']] exited with return code [20]
[YYYY-MM-DDTHH:MM:SS] ERROR      stderr: [Error 1021: Could not connect to the local service VMware AFD.
Verify VMware AFD is running.


• Reviewing systemctl status vmafdd.service shows unit vmafdd.service is masked
  

  # systemctl status vmafdd.service
Warning: The unit file, source configuration file or drop-ins of vmafdd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
 vmafdd.service
     Loaded: masked (Reason: Unit vmafdd.service is masked.)
    Drop-In: /etc/systemd/system.control/vmafdd.service.d
             └─50-BlockIOAccounting.conf, 50-CPUAccounting.conf, 50-MemoryAccounting.conf
             /etc/systemd/system/vmafdd.service.d
             └─dependency.conf
     Active: inactive (dead) since Day  [YYYY-MM-DDTHH:MM:SS] UTC; #min ago
        CPU: 43ms

  [YYYY-MM-DDTHH:MM:SS] vcsa.domain.local systemd[1]: Starting LSB: Authentication Framework Daemon...
[YYYY-MM-DDTHH:MM:SS] vcsa.domain.local vmafdd[2283]: Starting VMware afd Servicedone
[YYYY-MM-DDTHH:MM:SS] vcsa.domain.local systemd[1]: Started LSB: Authentication Framework Daemon.
[YYYY-MM-DDTHH:MM:SS] vcsa.domain.local systemd[1]: vmafdd.service: Succeeded.
[YYYY-MM-DDTHH:MM:SS] vcsa.domain.local systemd[1]: Stopped vmafdd.service.

Cause

This is due to several vCenter services being in a masked state.
A masked service is a systemd unit that has been completely disabled to prevent it from starting, either manually or automatically at boot, by symlinking its unit file to /dev/null.

 

Resolution

 Identify and unmask the affected services through the VCSA command line.

1. Connect to the vCenter server via SSH as root and run the following command to list the masked services
   systemctl list-unit-files | grep masked
   
   
2. To unmask all affected services at once, execute the following command:
   systemctl list-unit-files | grep masked | awk '{print $1}' | xargs -n1 systemctl unmask 
   
   
3.  Restart vCenter services
   service-control --stop --all && service-control --start --all
    
4.  Ensure all services are running post applying the fix:
   service-control --status--all