Firewall refresh causes disruptions to vSAN iSCSI services

Products

• VMware vSAN

Issue/Introduction

Examples of a firewall refresh causing disruption to vSAN iSCSI services.

Symptoms:
Examples of how vSAN iSCSI services can lose it's configuration.

• Running ESXi command 'esxcli network firewall refresh' on a ESXi host.
• Changing syslog location in vCenter
• Upgrading VC. This can push a refresh of ESXi firewall, in turn, disrupting vSAN iSCSI services.

esxupdate.0:2022-07-29T20:56:59Z esxupdate: msyslogd /etc/init.d/vmtoolsd /etc/init.d/hbr-agent
esxupdate.0:2022-07-29T20:56:59Z esxupdate: 13021993: LiveImageInstaller: INFO: Starting service /etc/init.d/vmware-fdm...
esxupdate.0:2022-07-29T20:56:59Z esxupdate: 13021993: vmware.runcommand: INFO: runcommand called with: args = '['/etc/init.d/vmware-fdm', 'start', 'upgrade']', outfile = 'None', returnoutput = 'True', timeout = '0.0'.
esxupdate.0:2022-07-29T20:56:59Z esxupdate: 13021993: LiveImageInstaller: DEBUG: Output: Not starting vmware-fdm now (upgrade). Will be started separately. success
esxupdate.0:2022-07-29T20:56:59Z esxupdate: 13021993: LiveImageInstaller: INFO: Executing post inst trigger : 'Firewall Refresh Trigger'
esxupdate.0:2022-07-29T20:56:59Z esxupdate: 13021993: LiveImageInstaller: INFO: Running firewall refresh...
esxupdate.0:2022-07-29T20:56:59Z esxupdate: 13021993: vmware.runcommand: INFO: runcommand called with: args = '/sbin/esxcli network firewall refresh', outfile = 'None', returnoutput = 'True', timeout = '0.0'. 

- Inspect the firewall rules by running from ESXi CLI:
esxcli network firewall ruleset list
This check will confirm if it's enabled in the first place.

Then run:
esxcli network firewall ruleset rule list
This is to check if the vit firewall rules are expected.

Environment

VMware vSAN 7.0.x

VMware vSAN 8.0.x

 

Cause

Running 'firewall refresh' unloads vSAN iSCSI network rules for the ESXi firewall, impacting vitd connection. This drops vSAN iSCSI services.

Resolution

Upgrade vCenter/ESXi to ESXi 7.0P10 and higher or 8.0P06 and higher

Workaround:
Running /etc/init.d/vitd restart on the target owner host, will load all firewall ruleset for vSAN iSCSI services.

Additional Information

Impact/Risks:
losing iSCSI connection to LUNs in vSAN. Can cause production impact.