No free inodes on /var partition of CPM - /var/spool/clientmqueue filled with huge amount of files

Products

• CPM
• XOS

Issue/Introduction

Services running on the CPM might log errors related to insufficient disk space. When attempting to write to disk, the following error might be returned: No space left on device. 

The df command does not reveal that disk space on the partition is utilized on 100%. However, the df -i command reveals that there are not any free inodes left.

Example:
df -i 
Filesystem             Inodes  IUsed   IFree IUse% Mounted on 
/dev/md5               1001920 26401   975519 3%   / 
/dev/md1               13056   31      13025  1%   /boot 
/dev/mapper/d1vg-lv1   256000  256000  0      100% /var 
none                   448971  2       448969 1%   /dev/shm 

Cause

On the Linux or Unix systems, free inodes are consumed for each file created. It might happen that even if there is enough disk space, there are not inodes left. This happens when a huge amount of small files exists on the filesystem.

On a CPM, the /var/spool/clientmqueue directory might contain a large number of files. 

Resolution

A solution, which will automatically clean the /var/spool/clientmqueue directory, has been added to the following XOS versions: 
XOS-9.6.8.0 
XOS-9.7.4.0 
XOS-10.0.1 
XOS-11.0.0 

Workaround

If there are a significant number of files under /var/spool/clientmqueue directory, they can be removed to free the inodes.

To automate this task, add the following line/entry into /var/spool/cron/postgres file:

15 */2 * * * rm /var/spool/clientmqueue/[qd]*