100% inode use causes error: "No space left on device"

Products

• VMware Tanzu Greenplum

Issue/Introduction

gpstart is unable to start the database due to one or more segments reporting the error:

No space left on device

From segment host, df -h reporting root directory only 69% used. However inode usage reached 100%.

df -i /
Filesystem      Inodes   IUsed IFree IUse% Mounted on
/dev/sda3      2015232 2015232     0  100% /

The inode usage has reached 100%
This issue is normally caused by too many files generated at one host.

Environment

Product Version: 5.17

Resolution

Run below script at the host with root account:

for i in /*; do echo $i; find $i |wc -l; done 

Once identified certain directory included big amount of files, then we can dig down to see which base directory caused the problem:

for i in /xxx/xxx/*; do echo $i; find $i |wc -l; done 

If the related directory's file can be removed, we can use below command to delete them

cd /xxx/xxx/; find . -type f | xargs rm -rf

At last, check inode usage again.

df -i /xxxx/

Note: Removing files from your segment data directory can leave the segment in an inoperable state and may cause permanent data loss. If you encounter this error and cannot free enough inodes outside of the segment data directories, please contact support.