Data Collector - Java process crashed and can't be restarted

Products

• CA Performance Management - Usage and Administration

Issue/Introduction

This is the scenario :

Today, the java process from our data collector stopped suddenly. I haven't been able to get this to work again. 

Rebooting the server or manually stopping and starting the dcmd process did not lead to success. 

There is always the following error message in the karaf.out, but we have enough space in the partition where the Data Collector is installed (/opt):

------------------------------

[[email protected]~]# tail -f /opt/CA/IMDataCollector/apache-karaf-4.3.3/data/karaf.out

!ENTRY org.eclipse.osgi 4 0 2023-02-13 10:10:26.628
!MESSAGE Error reading configuration: No space left on device
!STACK 0
java.io.IOException: No space left on device
        at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.base/java.io.File.createTempFile(Unknown Source)
        at org.eclipse.osgi.storagemanager.StorageManager.initializeInstanceFile(StorageManager.java:201)
        at org.eclipse.osgi.storagemanager.StorageManager.open(StorageManager.java:722)
.
.
.

[[email protected] ~]# df -h
Filesystem                 Size  Used Avail Use% Mounted on
sysfs                                0       0     0     -        /sys
proc                                 0       0     0     -        /proc
devtmpfs                    7,8G       0  7,8G  0%   /dev
tmpfs                          7,8G       0  7,8G  0%  /dev/shm
devpts                              0       0     0    - /dev/pts
tmpfs                           7,8G    8,9M  7,8G 1% /run
/dev/mapper/vg1-root       13G    521M   12G   5% /
/dev/mapper/vg1-usr       4,8G    2,6G  2,0G 56% /usr
/dev/sda1                 976M    184M  726M 21% /boot
/dev/mapper/vg1-opt        23G     16G  6,4G  71% /opt
/dev/mapper/vg1-tmp       6,8G     32M  6,4G  1% /tmp
/dev/mapper/vg1-var       5,8G    3,9G  1,7G 71% /var

------------------------------

Environment

Release : Any supported version

Cause

In this case, the number of inodes were exausted in the /opt partition in Linux. This can be checked by using the "df -ih" command:

------------------------------

[[email protected] ~]# df -ih
Filesystem             Inodes  IUsed  IFree IUse% Mounted on
.
.
/dev/mapper/vg1-opt   1507328  1058927  0    71%  /opt
.
.

------------------------------

Resolution

After deleting around 1.400.000 .log files present in the "/opt/CA/IMDataCollector/apache-karaf-4.3.3/dcdebug/On-Demand-Log" directory, the problem was solved:

------------------------------

[[email protected] ~]# df -ih
Filesystem             Inodes  IUsed    IFree IUse% Mounted on
.
.
/dev/mapper/vg1-opt   1507328  1058927  448401 71% /opt
.
.
------------------------------

Additional Information

https://knowledge.broadcom.com/external/article?articleId=11871