vCenter HA destroy attempt failed with the error "Failed to reload firewall"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server

Issue/Introduction

Symptoms:

• The VCHA destroy operation fails when executed through both the vSphere Client and the CLI command: vcha-destroy -f
• As a result of the failed VCHA destroy operation, vCenter Server services are unable to start.
• The following log entries are from /var/log/vmware/vcha/destroy-vcha.log

YYYY-MM-DDTHH:MM:SS ERROR destroy_vcha Failed to configure PostgreSQL:
YYYY-MM-DDTHH:MM:SS ERROR destroy_vcha ['Traceback (most recent call last):\n', ' File "/usr/sbin/vcha-destroy", line 226, in undoPostgresReplicationConfiguration\n   pgUtil.configure_access(False, VCHA_DATA_DIR)\n', ' File "/usr/lib/vmware-vcha/scripts/postgres_util.py", line 121, in configure_access
  raise InvokeCommandException(\'Failed to reload firewall\'),
'cis.exceptions.InvokeCommandException: {\n  "detail": [
{
"id": "install.ciscommon.command.errinvoke",
"translatable": "An error occurred while invoking external command : \'%(0)s\'",
"args": [        "Failed to reload firewall"\n      ],
"localized": "An error occurred while invoking external command : \'Failed to reload firewall\'"
} ],
  "componentKey": null,
  "problemId": null,
  "resolution": null}']
...
-A inbound -s <IP-Address>/<Subnet-Mask> -i eth0 -j return
...
YYYY-MM-DDTHH:MM:SS PM KST [49678]CRITICAL:firewall-reload:Cannot apply IPv4 firewall. ErrorCode=2 Output=b"iptables-restore v1.8.3 (legacy): 
Couldn't load target `return':No such file or directory\n\nError occurred at line: 17 Try `iptables-restore -h' or 'iptables-restore --help' for more information.\n"


Environment

vCenter Server 7.x

Cause

•  A firewall rule was created via the API for generating bulk entries:
  com.vmware.appliance.version1.networking.firewall.addr.inbound.add
  
  
• The rule was implemented using the following command:
  com.vmware.appliance.version1.networking.firewall.addr.inbound.add --pos 1 --prefix 32 --interface nic0 --address 10.x.x.x --policy 'return'


• In the vCenter server firewall configuration file under /etc/vmware/appliance/firewall.conf, the "policy" for the rule is set to lowercase "return". It should instead be uppercase "RETURN" to comply with expected configuration standards.
  
  
• The firewall rule API does not provide built-in validation. Therefore, users must perform manual validation of firewall rules prior to creation to ensure correctness. 

Resolution

Currently, there is no permanent resolution, as the issue is caused by a configuration mismatch in the environment.

As a workaround, the following steps can be implemented:

1. Identify the firewall rules with case-sensitive mismatches and correct them. Remove any incorrect entries prior to executing the VCHA destroy operation.

2. Firewall entries can be removed using the API command:
   com.vmware.appliance.version1.networking.firewall.addr.inbound.delete

     3. After removing all incorrect entries, attempt the VCHA destroy again:
          vcha-destroy -f