vpxa shuts down with error "Required property rule is missing from data object of type HostFirewallRuleset"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• An ESXi host  in a vSAN cluster is seen as "not responding" in vCenter
• Virtual machines continue to run and are not impacted
• You are able to SSH and login to Shell via DCUI
• You see the below error in the ESXi host vpxa.log

[YYYY-MM-DDTHH:MM:SS] error vpxa[9DE0B70] [Originator@6876 sub=vmomi.soapStub[1]]
--> Required property rule is missing from data object of type HostFirewallRuleset

-->
--> while parsing serialized DataObject of type vim.host.Ruleset

--> error parsing Any with xsiType HostFirewallRuleset

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7

Cause

The ESXi host  was in "Not responding" state in vCenter because the vpxa agent shut down due to an error while parsing an "empty firewall rule set"

In the vpxa.log you see entries similar to:

error vpxa[9DE0B70] [Originator@6876 sub=vmomi.soapStub[1]]
--> Required property rule is missing from data object of type HostFirewallRuleset
-->
--> while parsing serialized DataObject of type vim.host.Ruleset
--> error parsing Any with xsiType HostFirewallRuleset
--> at line 7, column 370
-->
--> while parsing property "val" of static type anyType
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.Change
--> at line 7, column 292
-->
--> while parsing property "changeSet" of static type ArrayOfPropertyChange
-->
--> while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectUpdate
--> at line 7, column 211
-->
--> while parsing property "objectSet" of static type ArrayOfObjectUpdate

error vpxa[8E7AB70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-41a7] Got error result from WaitForUpdates(): N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
--> [context]zKq7AUoCAgAAAGA8ZwAPdnB4YQAAv8UtbGlidm1hY29yZS5zbwAAMloSAA/QDQEOcw1saWJ2bW9taS5zbwABVDwNAbU/DQGpgg0B1VUNAhFZFnZweGEAAGM3IACOOyAAhYIgAMRCLgPKbABsaWJwdGhyZWFkLnNvLjAABE5JDWxpYmMuc28uNgA=[/context]
error vpxa[8E7AB70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-41a7] Fatal error while handling updates: N5Vmomi5Fault11SystemError9ExceptionE(vmodl.fault.SystemError)
--> [context]zKq7AUoCAgAAAGA8ZwAPdnB4YQAAv8UtbGlidm1hY29yZS5zbwAAMloSAA/QDQEOcw1saWJ2bW9taS5zbwABVDwNAbU/DQGpgg0B1VUNAhFZFnZweGEAAGM3IACOOyAAhYIgAMRCLgPKbABsaWJwdGhyZWFkLnNvLjAABE5JDWxpYmMuc28uNgA=[/context]
vpxa[8E7AB70] [Originator@6876 sub=vpxaInvtHostCnx opID=WFU-41a7] [VpxaInvtHost] Can't connect to hostd. Shutting down...
info vpxa[8E7AB70] [Originator@6876 sub=Default opID=WFU-41a7] [Vpxa] Shutting down now

Resolution

To resolve the issue, manually restart vpxa  by running the below command:

/etc/init.d/vpxa start

Additional Information

Refer to the below KB for steps to restart vpxa agent.

Restarting Management Agents in ESXi