[VMTP] Too many open files

Products

• CA Application Delivery Analysis MTP (NetQoS / ADA)

Issue/Introduction

Our VMTP has stopped processing data.

If we reboot the device, it will work temporarily but eventually end up in the same condition

Looking at the nqVerticaBulkLoad logs we see the following error:

./nqDBLoadIPSession.sh: cannot make pipe for command substitution: Too many open files

Environment

Release : 11.1

Cause

Vertica can have a very large amount of files open at one time.

Within linux there is a maximum configured by default that is fairly low.

When you hit that maximum, it will not allow additional files to be opened simultaneously.

Resolution

1) Verify the currently configured limit:

'ulimit -a'

The parameter we are concerned about is 'open files'. By default this is set to 1024 which is way too low.

 

2) Update this parameter and some related ones to better serve the amount of files Vertica requires:

Edit /etc/sysctl.conf add the below line, save and exit

fs.file-max = 100000

 

Edit /etc/security/limits.conf add the below lines before the #End, save and exit

* soft nproc 65535
 * hard nproc 65535
 * soft nofile 65535
 * hard nofile 65535

3) Verify the change by using 'ulimit -a' again.

Additional Information

https://www.vertica.com/docs/10.0.x/HTML/Content/Authoring/InstallationGuide/BeforeYouInstall/maxfiles.htm