Failed to start vcha services. Error: A system error occurred. Check logs for details.

Products

• VMware vCenter Server

Issue/Introduction

• VCHA set up fails at 90%
• 
• vpxd.log on active VC node:
  • yyyy-mm-ddthh:mm:ss.788+09:00 error vpxd[03292] [Originator@6876 sub=vpxLro opId=########-89a1-4c51-87a1-############] [VpxLRO] Unexpected Exception: N5Vmomi5Fault11SystemEr
    ror9ExceptionE(Fault cause: vmodl.fault.SystemError
    --> )
    --> [context]zKq7AVECAQAAAOIfcwEUdnB4ZAAAAto3bGlidm1hY29yZS5zbwAAmXksABdtLQAf6jIBgfBxdnB4ZAABRfJxgdIKKwGBOEMrAYH4SysBgUWHKwGB9QRpAYE2BmkBgWQVaQGBx0JoAYGS7GgBAOdJIwB1nyMAwGU3
    Aod/AGxpYnB0aHJlYWQuc28uMAADvzYPbGliYy5zby42AA==[/context]
    yyyy-mm-ddthh:mm:ss.791+09:00 error vpxd[03292] [Originator@6876 sub=FailoverClusterOperator opId=########-89a1-4c51-87a1-############] Exception: N5Vmomi11MethodFault9Excep
    tionE(Fault cause: vmodl.fault.SystemError
    --> )
    --> [context]zKq7AVECAQAAAOIfcwEPdnB4ZAAAAto3bGlidm1hY29yZS5zbwAAmXksABdtLQAf6jKBr3EtAXZweGQAgUEGaQGB+AhpAYGmFWkBgcdCaAGBkuxoAQDnSSMAdZ8jAMBlNwKHfwBsaWJwdGhyZWFkLnNvLjAAA782
    D2xpYmMuc28uNgA=[/context]
    yyyy-mm-ddthh:mm:ss.811+09:00 info vpxd[03292] [Originator@6876 sub=vpxLro opId=########-89a1-4c51-87a1-############] [VpxLRO] -- FINISH task-1989768
    yyyy-mm-ddthh:mm:ss.811+09:00 info vpxd[03292] [Originator@6876 sub=Default opId=########-89a1-4c51-87a1-############] [VpxLRO] -- ERROR task-1989768 -- Datacenters -- com.v
    mware.vcenter.vcha.cluster.deploy: vmodl.fault.SystemError:
    --> Result:
    --> (vmodl.fault.SystemError) {
    -->    faultCause = (vmodl.MethodFault) null,
    -->    faultMessage = <unset>,
    -->    reason = "Failed to start HACore profile on node <IP>"
    -->    msg = ""
    --> }
• On passive VC node at var/log/vmware/vcha/fixVchaConfigFiles.log:
  • yyyy-mm-ddthh:mm:ss.700Z ERROR fixVchaConfigFiles Error updating configuration
    Traceback (most recent call last):
      File "/usr/lib/vmware-vcha/scripts/fixVchaConfigFiles.py", line 131, in main
        updateConfiguration()
      File "/usr/lib/vmware-vcha/scripts/fixVchaConfigFiles.py", line 95, in updateConfiguration
        writePgFirewallOpenAccess(node1Ip)
      File "/usr/lib/vmware-vcha/scripts/fixVchaConfigFiles.py", line 32, in writePgFirewallOpenAccess
        rulesMap = json.load(f)
      File "/usr/lib/python3.7/json/__init__.py", line 296, in load
        parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
      File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    (END) 

Environment

• vCenter Server 7.x
• vCenter Server 8.x

Cause

• The script fixVxhaConfigFiles.py encounters failure while loading the file contents of /usr/lib/vmware-vcha/data/pg-firewall-block.conf into a JSON object as per the log snippets seen on passive node.
• /usr/lib/vmware-vcha/data/pg-firewall-block.conf is empty.
• The expected workflow is that the file /etc/vmware/appliance/firewall/vmware-vpostgres is copied to /usr/lib/vmware-vcha/data/pg-firewall-block.conf during VCHA configuration, and then used in fixVchaConfigFiles.py.
• Content of /etc/vmware/appliance/firewall/vmware-vpostgres
  {
    "internal-ports": {
      "rules": [
        {
          "name": "vpostgres.int.server_port",
          "port": 5432
        }
      ]
    }
  }

Resolution

• Workaround:
  • Destroy the VCHA and re-try the VCHA setup. 
  • Upon reaching the 80-85% setup, SSH in to the passive node and run the below command. 
    • cp /etc/vmware/appliance/firewall/vmware-vpostgres /usr/lib/vmware-vcha/data/pg-firewall-block.conf
  • Post this, the VCHA configuration will be successful.