NFS datastore mount operation fails with error "Failed to add new allowed ip, limit exceeded on number of allowed ip. Maximum allowed ip per ruleset is 128"
search cancel

NFS datastore mount operation fails with error "Failed to add new allowed ip, limit exceeded on number of allowed ip. Maximum allowed ip per ruleset is 128"

book

Article ID: 313455

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Allow to mount more than 128 NFS volumes on ESXi host.

Symptoms:

  • Failed to mount more than 128 NFS datastores.

  • Mount operation fails with error "Mount failed : xx.xx.xx.xx is neither an IPV4 address nor IPV6 address".

  • The allowed ip in firewall rule list have 128 unique IP's listed.

Sample output :

[root@xxxx-xxxx-xxxxx-ep-03:~] esxcli network firewall ruleset allowedip list| grep nfs

nfsClient         xx.xxx.2.18, xx.yyy.zzz.80, xx.yyy.zzz.126, xx.xxx.3.9, xx.yyy.zzz.78, xx.yyy.zzz.98, xx.yyy.zzz.99, xx.yyy.zzz.177, xx.xxx.2.46, xx.yyy.203.60, xx.yyy.203.77, xx.yyy.203.66, xx.yyy.zzz.178, xx.yyy.203.13, xx.yyy.zzz.198, xx.xxx.2.54, xx.yyy.zzz.152, xx.yyy.zzz.121, xx.yyy.zzz.185, xx.xxx.2.36, xx.yyy.zzz.235, xx.yyy.203.62, xx.yyy.zzz.240, xx.xxx.2.20, xx.yyy.203.73, xx.yyy.zzz.87, xx.xxx.2.63, xx.yyy.zzz.170, xx.yyy.zzz.79, xx.yyy.zzz.xx7, xx.xxx.2.2, xx.yyy.zzz.153, xx.yyy.zzz.215, xx.yyy.203.xx, xx.xxx.2.12, xx.yyy.zzz.120, xx.yyy.zzz.134, xx.xxx.2.44, xx.yyy.zzz.132, xx.yyy.zzz.66, xx.yyy.zzz.173, xx.xxx.2.32, xx.yyy.zzz.226, xx.yyy.zzz.73, xx.xxx.2.16, xx.yyy.203.80, xx.yyy.zzz.180, xx.yyy.zzz.183, xx.xxx.2.58, xx.yyy.203.29, xx.yyy.zzz.139, xx.yyy.zzz.227, xx.xxx.2.55, xx.yyy.zzz.154, xx.yyy.zzz.146, xx.yyy.zzz.84, xx.xxx.2.29, xx.yyy.203.33, xx.yyy.zzz.248, xx.xxx.2.19, xx.yyy.203.38, xx.yyy.zzz.91, xx.yyy.zzz.93, xx.xxx.2.69, xx.yyy.zzz.131, xx.yyy.zzz.174, xx.yyy.zzz.zzz, xx.yyy.zzz.82, xx.yyy.zzz.161, xx.yyy.zzz.83, xx.xxx.2.47, xx.yyy.zzz.199, xx.yyy.zzz.122, xx.yyy.203.40, xx.xxx.2.xx, xx.yyy.203.44, xx.yyy.zzz.182, xx.xxx.2.66, xx.yyy.203.22, xx.yyy.203.34, xx.yyy.zzz.207, xx.xxx.2.64, xx.yyy.203.71, xx.yyy.203.25, xx.xxx.2.52, xx.yyy.zzz.74, xx.yyy.203.72, xx.yyy.zzz.71, xx.xxx.2.45, xx.xxx.2.6, xx.yyy.zzz.165, xx.yyy.203.58, xx.yyy.zzz.200, xx.xxx.2.42, xx.yyy.zzz.218, xx.yyy.zzz.148, xx.yyy.203.47, xx.xxx.2.23, xx.yyy.zzz.225, xx.yyy.zzz.86, xx.yyy.203.4, xx.yyy.zzz.233, xx.xxx.2.26, xx.yyy.zzz.135, xx.yyy.203.69, xx.yyy.zzz.140, xx.xxx.2.57, xx.yyy.203.53, xx.115.249.168, xx.yyy.203.37, xx.yyy.zzz.231, xx.xxx.2.24, xx.yyy.zzz.192, xx.yyy.zzz.221, xx.yyy.zzz.xx3, xx.xxx.2.4, xx.yyy.203.54, xx.yyy.zzz.196, xx.xxx.2.48, xx.yyy.zzz.163, xx.xxx.2.3, xx.yyy.203.64, xx.yyy.zzz.220, xx.xxx.2.53, xx.yyy.zzz.xx0, xx.yyy.zzz.125, xx.yyy.203.48, xx.yyy.zzz.217

nfs41Client        All

 

Environment

VMware ESXi 6.5.x
VMware vSphere ESXi 7.x
VMware ESXi 6.7.x
VMware ESXi 8.0.x

 

Cause

The firewall entries are managed internally by NFS. NFS userworld subsytem takes care of adding/removing IP addresses from the list during mount and unmount operations. NFS uses IP address internally for mount operation, environments with more than 128 unique IP addresses used for each of NFS datastore will hit the hard limit of 128 in firewall.

Resolution

To resolve the issue, please follow the below mentioned steps:

1. For releases later than ESXi 7.0 U3 and prior to ESXi 8.0 U2:

  1. Place the ESXi host in maintenance mode

  2. Unmount all the NFS volumes

  3. Make sure that NFS associated IP's are not listed in command

esxcli network firewall ruleset allowedip list | grep nfs

  1. To allow beyond 128 unique IPs, users can specify a IP/mask configuration in firewall  for nfs Client ruleset, follow below commands

esxcli network firewall ruleset allowedip add -i xx.xx.xx.xx/24 -r nfsClient
esxcli network firewall ruleset allowedip add -i xx.xx.xx.xx/24 -r nfsClient

  1. Mount all the NFS datastores and make sure only single entry is created for allowed  ip's in command

esxcli network firewall ruleset allowedip list | grep nfs

  1. This settings is persistent across host reboot, you can confirm by rebooting the    host

Output from test environment
Steps with sample IP's :

# add an IP/mask entry for IPv4
[:~] esxcli network firewall ruleset allowedip add -i xx.xx.xx.xx/8 -r nfsClient
[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient     xx.xx.xx.xx/8
nfs41Client     All

# add the datastores
[:~] esxcli storage nfs add -H xx.xx.xx.xx -v nfs1 -s /store1
[:~] esxcli storage nfs add -H xx.xx.xx.xx -v nfs2 -s /store2

# no new entries are added, since the two IPs are part of the network
[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient     xx.xx.xx.xx/8
nfs41Client     All
Tested after reboot
[:~] reboot host to check if it's persistent

# entries are intact after reboot
[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient     xx.xx.xx.xx/8
nfs41Client     All

2. For releases from ESXi 8.0 U2:

To allow beyond 128 unique IP addresses, specify a IP address/mask configuration in firewall for nfsClient ruleset.

New esxcli commands has been added to add IP address and mask entry in the firewall rule set, to remove the IP address and mask entry and to list the entries as below:

# add an IP/mask entry
esxcli storage nfs(or nfs41) firewall add -F x.x.x.x/y

# remove an IP/mask entry
esxcli storage nfs(or nfs41) firewall remove -F x.x.x.x/y

# List the configured IP/Mask entries in nfsClient/nfs41Client firewall rulesets.
esxcli storage nfs(or nfs41) firewall list

The code fix also checks for such entries in the list of allowed IPs and skips adding a new entry for the IP if it is already part of the IP/mask network, thus bringing down the total number of firewall entries.

Sample output:

** Steps with sample IP's :

# add an IP/mask entry for IPv4
[:~] esxcli storage nfs firewall add -F xx.xx.xx.xx/8

[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient     xx.xx.xx.xx/8
nfs41Client     All

[:~] esxcli storage nfs firewall list
   Allowed IPMask: xx.xx.xx.xx/8

[:~] esxcli storage nfs41 firewall list
   Allowed IPMask:

# add the datastores
[:~] esxcli storage nfs add -H xx.xx.xx.xx -v nfs1 -s /store1
[:~] esxcli storage nfs add -H xx.xx.xx.xx -v nfs2 -s /store2

 # no new entries are added, since the two IPs are part of the network
[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient     xx.xx.xx.xx/8
nfs41Client     All

[:~] esxcli storage nfs firewall list
   Allowed IPMask: xx.xx.xx.xx/8

[:~] esxcli storage nfs41 firewall list
   Allowed IPMask:
 
Tested after reboot
[:~] reboot host to check if it's persistent

# entries are intact after reboot
[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient     xx.xx.xx.xx/8
nfs41Client     All

[:~] esxcli storage nfs firewall list
   Allowed IPMask: xx.xx.xx.xx/8

[:~] esxcli storage nfs41 firewall list
   Allowed IPMask:

# remove an IP/Mask entry
[:~] esxcli storage nfs firewall remove -F 1.1.1.1/30

[:~] esxcli storage nfs firewall list
   Allowed IPMask:
[:~] esxcli storage nfs41 firewall list
   Allowed IPMask:

[:~] esxcli network firewall ruleset allowedip list | grep nfs
nfsClient    
nfs41Client     All

Additional Information

Impact/Risks:
Cannot mount more than 128 NFS datastores where as maximum number of supported NFS volume is 256.

Powered by Wolken Software