vCenter HA Destroy operation failed with "Failed to reload firewall error"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server

Issue/Introduction

This article provides information on "destroying VCHA failure" due to vCenter firewall configuration.

Symptoms:

1. VCHA destruction fails using both the vSphere UI and the CLI command vcha-destroy -f.
2. An error message related to the VCHA destroy failure is displayed within the vSphere UI.
3. vCenter Server services are unable to start because of the failed VCHA destroy operation.
4. Similar log entries like below from destroy-vcha.log:

YYYY-MM-DDTHH:MM:SS ERROR destroy_vcha Failed to configure PostgreSQL:
YYYY-MM-DDTHH:MM:SS ERROR destroy_vcha ['Traceback (most recent call last):\n', ' File "/usr/sbin/vcha-destroy", line 226, in undoPostgresReplicationConfiguration\n   pgUtil.configure_access(False, VCHA_DATA_DIR)\n', ' File "/usr/lib/vmware-vcha/scripts/postgres_util.py", line 121, in configure_access\n
  raise InvokeCommandException(\'Failed to reload firewall\')\n',
'cis.exceptions.InvokeCommandException: {\n  "detail": [\n
{\n
"id": "install.ciscommon.command.errinvoke",\n
"translatable": "An error occurred while invoking external command : \'%(0)s\'",\n
"args": [\n        "Failed to reload firewall"\n      ],\n      
"localized": "An error occurred while invoking external command : \'Failed to reload firewall\'"\n
}\n  ],
\n  "componentKey": null,
\n  "problemId": null,
\n  "resolution": null\n}
\n']

...
-A inbound -s 10.11.41.57/32 -i eth0 -j return
...
YYYY-MM-DDTHH:MM:SS PM KST [49678]CRITICAL:firewall-reload:Cannot apply IPv4 firewall. ErrorCode=2 Output=b"iptables-restore v1.8.3 (legacy): 
Couldn't load target `return':No such file or directory\n\nError occurred at line: 17\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n"

Environment

VMware vCenter Server 7.0.x

Cause

1. A firewall rule was created using the API for generating bulk quantities.
    com.vmware.appliance.version1.networking.firewall.addr.inbound.add

 2. The rule was created using the following command:
    com.vmware.appliance.version1.networking.firewall.addr.inbound.add --pos 1 --prefix 32 --interface nic0 --address 10.x.x.x --policy 'return'

 3. As per the above rule in the following firewall configuration file of the vCenter server, "policy": "return" is configured with lower case, however it should be configured with Upper case "RETURN"
    /etc/vmware/appliance/firewall.conf

 4. The firewall rule API does not include validation capabilities. Consequently, users are required to perform manual validation prior to rule creation.

Resolution

Currently there is no resolution since it is a configuration problem in the environment.

As a workaround the following can be performed:

1. Check the rules which has mismatch in case sensitive and modify those rules. Remove those entries before executing destroy VCHA command.
   
   
2. Can use API to remove entries:
   com.vmware.appliance.version1.networking.firewall.addr.inbound.delete

     3. Destroy VCHA again after removing all wrong entries.
          vcha-destroy