Event burst for 'esx.audit.net.firewall.config.changed' and vCenter storage/seat exhaustion.

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server /storage/seat partition fills rapidly (e.g., reaching 96%+).
• vCenter event logs show thousands entries per day for esx.audit.net.firewall.config.changed per ESXi.
• ESXi /var/run/log/hostd.log shows continuous addIP4 and removeIP4 operations for the nfsClient firewall ruleset.
  
  Firewall configuration has changed. Operation 'addIP4' for rule set nfsClient succeeded.
Firewall configuration has changed. Operation 'removeIP4' for rule set nfsClient succeeded.
Firewall configuration has changed. Operation 'addIP4' for rule set nfsClient succeeded.
Firewall configuration has changed. Operation 'removeIP4' for rule set nfsClient succeeded.

Environment

vSphere 8.0

Cause

ESXi nfsClient firewall is a system-owned service that dynamically whitelists only the specific IP addresses associated with active NFS connections. This issue occurs when an ESXi host attempts to communicate with a stale or orphaned NFS datastore reference that is no longer reachable.

When an ESXi host resolves the NFS server FQDN, it sequentially attempts connectivity to each resolved IP address. For unreachable IPs, the host repeatedly:

• Adds the IP address to the NFS client firewall ruleset
• Tests connectivity
• Removes the IP address if connectivity fails

Resolution

Identify and remove any orphaned or decommissioned NFS mounts that may be contributing to connection retry loops. 

1. Verify the list of currently mounted NFS datastores to determine if the problematic mount is visible to the management agents:
   esxcli storage nfs list
2. vSphere environments, a datastore may be absent from the as a standard mount but remain registered in the configuration store. This results in persistent connection attempts and log noise.
   
   
3. Retrieve the current NFS configuration from the store:
   configstorecli config current get -c esx -g storage -k nfs_v3_datastores


4. Once the orphaned datastore name is identified from either of the steps above, execute the following command to unmount the volume and remove the configuration reference:
   esxcli storage nfs remove -v <Datastore_Name>