"Stopped services: 'vcha' error" while running a VAMI backup

Products

• VMware vCenter Server

Issue/Introduction

Fixing the VAMI backup which is failing due to the vcha service

Symptoms:
The vCenter Server VAMI Backup fails with error:

• Invalid vCenter Server Status: All required services are not up! Stopped services: 'vcha'

Manually starting vcha fails with the following error

• Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vcha services. Error: Service crashed while starting

Getting the following logs in /var/log/vmware/applmgmt/backupScheduler.log

YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-37026] [Scheduler::ExecScheduleRun:Scheduler.py:138] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.applmgmt.err_invalid_vc_status', default_message="Invalid vCenter Server Status: All required services are not up! Stopped services: 'vcha'.", args=['vcha'], params=None, localized=None)], data : None, error_type : None}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1171, in run
    'comment': comment,
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke
    self._rest_converter_mode)
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.applmgmt.err_invalid_vc_status', default_message="Invalid vCenter Server Status: All required services are not up! Stopped services: 'vcha'.", args=['vcha'], params=None, localized=None)], data : None, error_type : None} 

YYYY-MM-DDTHH:MM:SS  [0] [MainProcess:PID-38330] [Scheduler::ExecArchiveList:Scheduler.py:120] ERROR: Failed to issue the Archive.list request. Exception: {messages : [LocalizableMessage(id='vapi.method.invoke.exception', default_message="Error in method invocation 'NoneType' object is not subscriptable", args=["'NoneType' object is not subscriptable"], params=None, localized=None)], data : None, error_type : INTERNAL_SERVER_ERROR}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 116, in ExecArchiveList
    backupList = svc_handle.list(locationSpec, systemName, filterSpec)
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup/system_name_client.py", line 289, in list
    'filter_spec': filter_spec,
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke
    self._rest_converter_mode)
com.vmware.vapi.std.errors_client.InternalServerError: {messages : [LocalizableMessage(id='vapi.method.invoke.exception', default_message="Error in method invocation 'NoneType' object is not subscriptable", args=["'NoneType' object is not subscriptable"], params=None, localized=None)], data : None, error_type : INTERNAL_SERVER_ERROR}

Environment

VMware vCenter Server 7.0.3

Cause

The startup type for vcha service is set to automatic, which should not be the case if VCHA is not configured in the environment.

Resolution

1. Verify that the customer has not configured VCHA in his environment.
2. Run the command to check start type of the vcha service /usr/lib/vmware-vmon/vmon-cli --status vcha 
   1.  You will receive an ouput similar to this
      1. Name: vcha
         Starttype: AUTOMATIC
         RunState: STOPPED
         RunAsUser: root
         CurrentRunStateDuration(ms): 67159080
         HealthState: UNHEALTHY
         FailStop: TRUE
         MainProcessId: N/A
3. Run this command to disable VCHA service /usr/lib/vmware-vmon/vmon-cli -S DISABLED -U vcha
4. Run this command to check the Startup type which should change to disabled /usr/lib/vmware-vmon/vmon-cli --status vcha
5. Manually Reboot the vCenter using the command, reboot
6. After rebooting once all the service's are up and running check the startup type again with the same command.
7. Run the VAMI backup again and it should not complain about the vcha service.

Additional Information

Impact/Risks:
Manual and Scheduled VAMI backup fails.