VMs lose connection to the vSAN iSCSI targets when the syslog configuration is modified

Products

• VMware vSAN

Issue/Introduction

Symptoms:

Initiator VMs can lose connection to VSAN iSCSI targets.

In ESXI log files you might see following events:

In hostd.log:
2024-10-09T15:52:47.605Z In(166) Hostd[2103950] [Originator@6876 sub=Hostsvc.SyslogConfigProvider opID=27fa81f1-76-632f sid=523212a0 user=vpxuser:domain.local\admin] Set called with key 'Syslog.global.logHost', value '"udp://10.xx.xx.10:514,udp://10.xx.xx.5:514"'
2024-10-09T15:52:47.608Z In(166) Hostd[2103953] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    key = 200,
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    chainId = -1295546496,
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    createdTime = "1970-01-01T00:00:00Z",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    userName = "",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    host = (vim.event.HostEventArgument) {
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->       name = "XXXXXXX",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->       host = 'vim.HostSystem:ha-host'
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    },
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    eventTypeId = "esx.audit.net.firewall.config.changed",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    arguments = (vmodl.KeyAnyValue) [
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->       (vmodl.KeyAnyValue) {
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->          key = "1",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->          value = "remove"
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->       },
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->       (vmodl.KeyAnyValue) {
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->          key = "2",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->          value = "dynamicruleset"
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->       }
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    ],
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    objectId = "ha-host",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] -->    objectType = "vim.HostSystem",
2024-10-09T15:52:47.609Z In(166) Hostd[2103905] --> }

In vobd.log:
2024-10-09T15:52:47.608Z In(14) vobd[2097867]  [netCorrelator] 6229076319782us: [vob.net.firewall.config.changed] Firewall configuration has changed. Operation 'remove' for rule set dynamicruleset succeeded.
2024-10-09T15:52:47.608Z In(14) vobd[2097867]  [netCorrelator] 6228989930015us: [esx.audit.net.firewall.config.changed] Firewall configuration has changed. Operation 'remove' for rule set dynamicruleset succeeded.

In  vmkernel.log:
2024-10-09T15:52:51.541Z Wa(180) vmkwarning: cpu93:2102898)WARNING: 10.xx.xx.40 (iqn.XXXX.com.XXXX:xxxx.local): no ping reply (NOP-Out) after 5 seconds; dropping connection
2024-10-09T15:52:51.541Z Wa(180) vmkwarning: cpu93:2102898)WARNING: 10.xx.xx.41 (iqn.XXXX.com.XXXX:xxxx.local): no ping reply (NOP-Out) after 5 seconds; dropping connection

Environment

VMware ESXi 8.0.U2

VMware ESXi 8.0.U3

VMware ESXi 7.0.U3

Cause

When a syslog loghost is configured with a non-default port, a dynamic firewall rule is automatically created to allow connections to the loghost.
Upon removal of this loghost configuration, the associated dynamic rule is also deleted.
The current implementation removes all dynamic firewall rules that do not match the current loghost configuration, which is incorrect, as it also removes non-syslog rules.

 

Resolution

The fix will be available in  ESXi 8.0 U3 P05 and ESXi 7.0 U3 P10.

Workaround 

To restore vSAN ISCSI connectivity loss in ESXI host, run the below commands:

# esxcli network firewall set --default-action true

# /etc/init.d/vitd status
# /etc/init.d/vitd restart