NFS server's IP is removed automatically from firewall allowed list after esxi host rebooting on 8.0U2 or above due to bug

Products

• VMware vSphere ESXi

Issue/Introduction

This article is for informing known issue and providing workaround for adding ip allowed list

Symptoms:

• When using ESXi 8.0U2 or above build.
• Host suddenly can't mount NFS server after rebooting esxi host
• The nfsclient firewall is configured with 'noallowall' option.

2024-03-06T16:11:21.351Z In(14) vobd[2098027]:  [netCorrelator] 49648463us: [esx.audit.net.firewall.config.changed] Firewall configuration has changed. Operation 'enable' for rule set nfsClient succeeded.
2024-03-06T16:11:21.353Z In(14) vobd[2098027]:  [netCorrelator] 49650038us: [vob.net.firewall.config.changed] Firewall configuration has changed. Operation 'noallowall' for rule set nfsClient succeeded.

• A check of the nfs server connection using nc command then it fails
• From ip connection list, the status is in 'SYN_SENT'

tcp         0       0  192.X.X.X:927    192.X.X.X:111    SYN_SENT     10514960  newreno  RPC-tx-

• ​​​​There is no ip allowed list in firewall section from vsphere UI on this host which value was previously existed.
• Attempts to input NFS server ip in allowed ip list using vSphere UI or esxcli, will fail with error like below.

# esxcli network firewall ruleset allowedip add -i 10.X.X.X -r nfsClient

Invalid operation requested: Can not change allowed ip list this ruleset, it is owned by system service.

Environment

VMware vSphere ESXi 8.0.2
VMware vSphere ESXi 8.0

Cause

There are 2 issues which is related with this problem.

• Missing nfs allowed ip list after rebooting esxi host.

There are known issue(bug).

While ESX boot, if remount operation fails because of some reason, NFSv3 VSI interface returns exception and vmkctl removes server IP from nfsclient firewall rulset. 
 It is a regression due to the non-persistence of the system owned firewall.

• Failed to input allowed ip address in firewall ruleset using UI

From ESXi 8.0U2, some firewall ruleset is changed to 'systemd-owned' by default and prevented manual updating. nfsClient is one of them

Resolution

For missing nfs allowed ip list after rebooting esxi host, this fix will include in 8.0U3 release

Workaround:
Here is the alternative command for changing allowed ip list in firewall ruleset for nfsClient
- Adding ip (change IP address to correct one)

esxcli storage nfs firewall add --ip-mask=192.X.X.X/32

- Removing ip (change IP address to correct one)

esxcli storage nfs firewall remove --ip-mask=192.X.X.X/32

Additional Information

Impact/Risks:
NFS filesystem can't remount after rebooting esxi host.