Disk partition appears to be full even after removing large files

Products

• XOS

Issue/Introduction

Partition could appear to be full even after removing all large filesThere is no free space on the disk partition, even though all large files were removed. In all examples in this article we suppose that the /var partition appears to be full.

- Running the df -h command shows the /var partition to be 100% full

# df -h
Filesystem           Size Used Avail Use% Mounted on 
...
/dev/mapper/d1vg-lv1 2.0G 2.0G 2.0G  100% /var 
...

-  Running du -sh */ command in the /var directory does not show the space being used.
-  No large file could be found when running ls -l command

Cause

This issue may occur if an active process is using a deleted file. Such a  file in use is blocked by the process and occupies disk space even if it appears to be deleted.

Resolution

Use the lsof command to determine if some process is using the deleted files and then stop that process:

# lsof /var
...

cbsalarmm 3777 root 13w REG 253,1 744056352 64037 /var/log/alarm_xmlrpc.log (deleted)
...

If you see any file with (deleted) at the end, it is most likely the file which is blocked by some process.
 

Check to see if there is a daemon associated with the PID, such as cbsalarmmgrd. For example if you see cbsalarmmgrd blocking the file, restart the cbsalarmmgrd service:

# service cbsalarmmgrd restart

 

In case you can't restart the service, note the PID of the process and stop the process by running the kill command:

 

# kill -9 <pid>

Please always verify if it is safe to reload the service or kill the process blocking the file. If not sure please contact Crossbeam Technical Support. 

Alternatively, reloading the module will free up the blocked files as well.

Workaround

NA