Our VMTP has stopped processing data.
If we reboot the device, it will work temporarily but eventually end up in the same condition
Looking at the nqVerticaBulkLoad logs we see the following error:
./nqDBLoadIPSession.sh: cannot make pipe for command substitution: Too many open files
Release : 11.1
Vertica can have a very large amount of files open at one time.
Within linux there is a maximum configured by default that is fairly low.
When you hit that maximum, it will not allow additional files to be opened simultaneously.
1) Verify the currently configured limit:
The parameter we are concerned about is 'open files'. By default this is set to 1024 which is way too low.
2) Update this parameter and some related ones to better serve the amount of files Vertica requires:
Edit /etc/sysctl.conf add the below line, save and exit
fs.file-max = 100000
Edit /etc/security/limits.conf add the below lines before the #End, save and exit
* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535
3) Verify the change by using 'ulimit -a' again.