VAMI filed-based backup fails with "All required services are not up! Stopped services: 'vmcam'."

Products

• VMware vCenter Server

Issue/Introduction

• Scheduled VAMI file-based backup fails, both when attempted manually as well as scheduled
• When reviewing the backupscheduler.log in /var/log/vmware/applmgmt/, the following messages are seen:

<timestamp> [0] [MainProcess:PID-62363] [Scheduler::ExecScheduleRun:Scheduler.py:138] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.applmgmt.err_invalid_vc_status', default_message="Invalid vCenter Server Status: All required services are not up! Stopped services: 'vmcam'.", args=['vmcam'], params=None, localized=None)], data : None, error_type : None}
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.applmgmt.err_invalid_vc_status', default_message="Invalid vCenter Server Status: All required services are not up! Stopped services: 'vmcam'.", args=['vmcam'], params=None, localized=None)], data : None, error_type : None}
<timestamp> [0] [MainProcess:PID-13490] [Scheduler::ExecScheduleRun:Scheduler.py:138] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.applmgmt.err_invalid_vc_status', default_message="Invalid vCenter Server Status: All required services are not up! Stopped services: 'vmcam'.", args=['vmcam'], params=None, localized=None)], data : None, error_type : None}
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.applmgmt.err_invalid_vc_status', default_message="Invalid vCenter Server Status: All required services are not up! Stopped services: 'vmcam'.", args=['vmcam'], params=None, localized=None)], data :
None, error_type : None}

• The vmcamd logs in /var/log/vmware/vmcamd don't have any recent entries
• Looking the the vmon.log in /var/log/vmware/vmon, messages can be found about vmcam startup failed with the error code 5 (ERROR_ACCESS_DENIED).

 

Environment

VMware vCenter Server 7.0.x

Cause

This issue can occur if the vmcam user is lacking permission to write into the vmcamd-syslog.log.

This is usually due the file not being owned by the vmcamd user, but by a different account, for example root.

Resolution

To fix this problem,

1. Verify the current ownership of the log file by running the following command and looking for vmcamd-syslog.log. The file ownership will be root:root.

    # ls -lht /var/log/vmware/vmcamd/

2. Update the ownership to "vmcam" user using the command

   # chown vmcam:vmcam /var/log/vmware/vmcamd/vmcamd-syslog.log

3. Start vmcam service:

   # vmon-cli -i vmcam

 

Post this vmcamd should be in running state and the backup jobs should be successful.