Firewall network refresh or running "esxcli network firewall refresh" makes "iSCSI" ruleset disabled, when the UserValue in the config store is set to false, which impacts the iscsi connection to the storage.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• The iscsi firewall rule set can be disabled after a network firewall refresh part of vib update.

 

• The issue has been seen to occur after vcenter upgrade or vsphere replication upgrade, starting a network firewall refresh task which changes the iSCSI rule set configurations to 'disabled'.

 

• In the vobd logs you can see events similar to the logs below:
      [vob.net.firewall.config.changed] Firewall configuration has changed. Operation 'disable' for rule set iSCSI succeeded.
      [esx.audit.net.firewall.config.changed] Firewall configuration has changed. Operation 'disable' for rule set iSCSI succeeded.
      
      
• As a result the iscsi connections stop and causing an APD event on the datastores utilizing the iscsi connection.
  
   [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba68:C6:T0
   [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba68:C3:T0
   
   [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.xxxxx degraded. Path vmhba68:C6:T0:L1 is down. Affected datastores: "DATASTORE"
   [vob.scsi.scsipath.pathstate.dead] scsiPath vmhba68:C6:T0:L2 changed state from on
   [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.xxxxx degraded. Path vmhba68:C6:T0:L2 is down. Affected datastores: "DATASTORE".
    
   [vob.storage.apd.start] Device or filesystem with identifier [naa.xxxxx] has entered the All Paths Down state. 

 

• The firewall status changes to 'disable' and remains the same until the host is rebooted.

Cause

• During ESX host bootup. "iSCSI" ruleset is enabled by kmxa during jumpstart in the configstore autoconf stage.
  Per the design of configstore, during autoconf stage, it is not allowed to write configuration to user configure column. As a result, "iSCSI" ruleset is enabled in vmkernel and saved to AutoConfValue column of configure store:
  AutoConfValue = {"name": "iSCSI", "enabled": true} 

 

• If the user manully disable the "iSCSI" on VC UI or with esxcli, the UserValue column of the configure store updated to:
  UserValue = {"name": "iSCSI", "enabled": false}

 

• As a result when running "esxcli network firewall refresh" the "iSCSI" ruleset will be disabled, because the command will initiate the configstore object generation, which will merge the AutoConfValue, UserValue and VitalValue to a single object. The UserValue is in the end of the merge sequece, so the configstore object returned by configstore lib have "enabled"=false.  

 

 

 

Resolution

Note:
On 7.X, issue fixed on 7.0.3-p09 recently.
On 8.X, starts from 8.0.2, the issue is fixed.

Workaround:

To avoid any re-occurrence of the issue, the UserValue has to be modified to "enabled"=true by following the steps below:

1. Connect to the esxi host through ssh.

2. Create a script file "workaround.py" with the following content:
     import libconfigstorepy as cslib
     import json
     cs = cslib.ConfigStore.GetStore()
     csoId = cslib.ConfigStoreObjectId('firewall_rule_sets', 'network', 'esx')
     csoId.instanceId = 'iSCSI'
     iSCSICso = cs.Get(csoId)
     if iSCSICso is not None:
     iSCSICso.SetValue("enabled", True)
     print("Update iSCSI cso with: %s" % json.dumps(json.loads(iSCSICso.GetData()), indent=4))
     cs.Set(iSCSICso)

3. Run the script using the command below:
    python workaround.py

Additional Information

Impact/Risks:

Applying the workaround doesn't impact the production.