Post marking a node as standalone using vcha-set-node-standalone and destroying vcha,VCSA backup is failing

Products

• VMware vCenter Server

Issue/Introduction

• When starting a file-based backup using the vCenter Appliance Management Interface (VAMI) UI, the backup immediately fails with a message

Invalid vCenter Server Status: All required services are not up! Stopped services 'vcha, vsan-dps'

• attempts to start the services fail as follows:

  root@<vCenter>[ /var/log/vmware/applmgmt ]# service-control --start vmware-vcha
  
  Operation not cancellable. Please wait for it to finish...
  Performing start operation on service vcha...
  Error executing start on service vcha. Details {
      "componentKey": null,
      "problemId": null,
      "detail": [
          {
              "localized": "An error occurred while starting service 'vcha'",
              "translatable": "An error occurred while starting service '%(0)s'",
              "args": [
                  "vcha"
              ],
              "id": "install.ciscommon.service.failstart"
          }
      ],
      "resolution": null
  }
  Service-control failed. Error: {
      "componentKey": null,
      "problemId": null,
      "detail": [
          {
              "localized": "An error occurred while starting service 'vcha'",
              "translatable": "An error occurred while starting service '%(0)s'",
              "args": [
                  "vcha"
              ],
              "id": "install.ciscommon.service.failstart"
          }
      ],
      "resolution": null
  }
  root@<vCenter> [ /var/log/vmware/applmgmt ]#
  
  oot@<vCenter> [ /var/log/vmware/applmgmt ]# service-control --start vsan-dps
  Operation not cancellable. Please wait for it to finish...
  Performing start operation on service vsan-dps...
  Error executing start on service vsan-dps. Details {
      "detail": [
          {
              "translatable": "An error occurred while starting service '%(0)s'",
              "localized": "An error occurred while starting service 'vsan-dps'",
              "id": "install.ciscommon.service.failstart",
              "args": [
                  "vsan-dps"
              ]
          }
      ],
      "resolution": null,
      "componentKey": null,
      "problemId": null
  }
  Service-control failed. Error: {
      "detail": [
          {
              "translatable": "An error occurred while starting service '%(0)s'",
              "localized": "An error occurred while starting service 'vsan-dps'",
              "id": "install.ciscommon.service.failstart",
              "args": [
                  "vsan-dps"
              ]
          }
      ],
      "resolution": null,
      "componentKey": null,
      "problemId": null

  
  
  

Environment

VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 7.0.x

Cause

When vMon finds a service in stopped status, it will try verify that the stopped status is not caused by a failure. To validate this, vMon checks that the service parameter  FailStop is set to "False". If this parameter for any reason is stuck in "True", the issue occurs.

Resolution

To work around this issue, 

• Update the startup of type of services to the following:

  vmon-cli --update vcha --starttype DISABLED
  vmon-cli --update vmcam --starttype MANUAL
  vmon-cli --update vsan-dps --starttype MANUAL

• Verify that the status of these three services is correctly set and that FailStop is set to "false":

  vmon-cli --status vcha
  vmon-cli --status vsan-dps
  vmon-cli --status vmcam

• If the output of any the above command has the FailStop param as "True", follow up with a restart of all vCenter services:

  service-control --stop --all && service-control --start --all

• Get the above service status again and verify tat FailStop is "false"
• Re-attempt the file-based backup