vCenter services not starting automatically after a restore from an image based backup.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter / PSC not fully starting after restoring VM from backup.
• Backup method was an Image Based Backup.
• Backups using the VAMI file based backup are not impacted.
• vmon service is masked - thereby not allowing all services managed by vmon to start.
• Unmasking vmon allows services to start successfully.
• On reboot vmon becomes masked again.

On a manual start of the services, it fails as below,
==================================================
#service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Service lwsmd startup type is not automatic. Skip
Performing start operation on service vmafdd...
Service vmafdd startup type is not automatic. Skip
Performing start operation on service vmdird...
Service vmdird startup type is not automatic. Skip
Performing start operation on service vmcad...
Service vmcad startup type is not automatic. Skip
Performing start operation on service vmware-sts-idmd...
Service vmware-sts-idmd startup type is not automatic. Skip
Performing start operation on service vmware-stsd...
Service vmware-stsd startup type is not automatic. Skip
Performing start operation on service vmdnsd...
Service vmdnsd startup type is not automatic. Skip
Performing start operation on profile: ALL...
Service vmware-vmon startup type is not automatic. Skip
Service-control failed. Error: Failed to start services in profile ALL. RC=11, stderr=Connect error. ErrCode: 111
 

Here is how you check if vmon is masked
=======================================
# systemctl list-unit-files | grep vmware-vmon.service
vmware-vmon.service          masked

To unmask the service
=======================================
#systemctl unmask vmware-vmon.service

Note- this will allow you to start the service - but the issue will reoccur on reboot.

Environment

VMware vCenter Server Appliance 6.7.x

Resolution

Run reconciliation operations below after you restore the vCenter appliance from an image based backup.
 

1. Navigate to the vCenter Server Management Interface, https://vCenter_FQDN_or_IP:5480 , and click "Set up vCenter Server Appliance" button.
    

 

2. Input the root password and click [LOG IN].
   
   
    

3. Click [NEXT].
   
   
    

4. Input SSO user name and password, then click [FINISH].
   
   
    

5. Reconciliation starts. Wait 15 - 30 minutes to finish.
   
   
    

6. Reconciliation completes and you will be able to access vSphere Client.
   
   
    

 

Additional Information

"Failed to execute operation: Unit file is masked" when we try to start the service on VCSA 6.5
File-Based Backup and Restore of vCenter Server Appliance 
VMware vCenter Server Appliance 6.7 File-Based Backup & Restore (VIDEO) 
vCenter Server Appliance 6.7 File-Based Backup and Restore Walkthroughs (VMWARE BLOG)