Jobs are failing and the CA WA System Agent, although appears to be running, repeatedly logs "No space left on device" errors. However, after validating this with the df -k command, sufficient free space exists.
Excerpt from defaultlog_agent.log:
01/16/2018 18:04:48.332 EST-0500 1 Core Mailbox.outbound.CybFileService.openRandomAccessFile[:151] - java.io.FileNotFoundException: /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/database/queue_communicator.tmp (No space left on device)
at java.io.RandomAccessFile.open0(Unknown Source)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at cybermation.library.services.CybFileService.openRandomAccessFile(CybFileService.java:145)
at cybermation.library.collection.CybObjectList.cleanList(CybObjectList.java:283)
at cybermation.library.collection.CybObjectList.gc(CybObjectList.java:707)
at cybermation.library.collection.CybObjectQueue.get(CybObjectQueue.java:179)
at cybermation.library.collection.CybQueue.get(CybQueue.java:126)
at cybermation.agent.mailbox.CybMessageDistributorOutgoing.run(CybMessageDistributorOutgoing.java:353)
at java.lang.Thread.run(Thread.java:745)
Excerpt from the nohup.stderr file:
java.io.FileNotFoundException: log/messagedistributorincoming.log (No space left on device)
java.io.FileNotFoundException: log/messagedistributorincoming.log (No space left on device)
...
...
Entry: Persistence exception. Retrying:
cybermation.library.persistence.CybPersistenceException: cleanList failed.
...
...
Caused by: java.io.FileNotFoundException: /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/database/queue_inbox.tmp (No space left on device)
The filesystem in which the CA WA System Agent is installed had exhausted the maximum allocation of inodes. "inode" (index node) is a metadata block that contains information (size, type, etc) about a file system object (file, directory, link, etc). There is one inode per file system object. In many file systems, the inode limit is fixed and cannot be increased once filesystem is created.
To check the state of the inodes on a Linux system, use command: df -Ti
Example:
# df -Ti /opt/CA/
Filesystem Type Inodes IUsed IFree IUse% Mounted on
/dev/mapper/appsvg-lv00 ext3 327680 327680 0 100% /opt/CA
Verify the system has free inodes available with command df -i. If IUse% shows 100%, then the following options can be explored:
# find /opt/CA/ -type d -exec sh -c 'ls -1 "{}"|wc -l' \; -print | sed '$!N;s/\n/ /' | sort -n
From the output, remove files from the directory that has the most number of files. Deleting files result in freeing up their associate inodes.
Once you have increased the free inodes, restart the CA WA System Agent and check if the jobs are running successfully.
The CA WA System Agent spool logs may over a period of time accumulate and use up inodes. It is important to ensure the spool directory is maintained (housekeeping) periodically.