Jobs are failing and the System Agent logs are repeatedly logging "No space left on device" errors, while sufficient disk space is still available.

Products

• Workload Automation Agent

Issue/Introduction

 

Jobs are failing and the CA WA System Agent, although appears to be running, repeatedly logs "No space left on device" errors. However, after validating this with the df -k command, sufficient free space exists.

Excerpt from defaultlog_agent.log:

01/16/2018 18:04:48.332 EST-0500 1 Core Mailbox.outbound.CybFileService.openRandomAccessFile[:151] - java.io.FileNotFoundException: /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/database/queue_communicator.tmp (No space left on device)
at java.io.RandomAccessFile.open0(Unknown Source)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at cybermation.library.services.CybFileService.openRandomAccessFile(CybFileService.java:145)
at cybermation.library.collection.CybObjectList.cleanList(CybObjectList.java:283)
at cybermation.library.collection.CybObjectList.gc(CybObjectList.java:707)
at cybermation.library.collection.CybObjectQueue.get(CybObjectQueue.java:179)
at cybermation.library.collection.CybQueue.get(CybQueue.java:126)
at cybermation.agent.mailbox.CybMessageDistributorOutgoing.run(CybMessageDistributorOutgoing.java:353)
at java.lang.Thread.run(Thread.java:745)

 

Excerpt from the nohup.stderr file:

java.io.FileNotFoundException: log/messagedistributorincoming.log (No space left on device)
java.io.FileNotFoundException: log/messagedistributorincoming.log (No space left on device)

...

...

Entry: Persistence exception. Retrying:
cybermation.library.persistence.CybPersistenceException: cleanList failed.

...

...

Caused by: java.io.FileNotFoundException: /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/database/queue_inbox.tmp (No space left on device)

Environment

CA Workload Automation System Agent 11.3+ on Linux and UNIX.

Cause

 

The filesystem in which the CA WA System Agent is installed had exhausted the maximum allocation of inodes. "inode" (index node) is a metadata block that contains information (size, type, etc) about a file system object (file, directory, link, etc).  There is one inode per file system object. In many file systems, the inode limit is fixed and cannot be increased once filesystem is created.

To check the state of the inodes on a Linux system, use command: df -Ti

Example:

# df -Ti /opt/CA/
Filesystem                 Type    Inodes IUsed  IFree  IUse%  Mounted on
/dev/mapper/appsvg-lv00    ext3    327680 327680 0      100%   /opt/CA

Resolution

 

Verify the system has free inodes available with command df -i.  If IUse% shows 100%, then the following options can be explored:

• In case the filesystem affected is a logical volume, contact the System Admin to extend/increase the volume size.
• In case the filesystem is non-extendable, execute the following command to identify directories that contain most number of files (sorted ascending):

        # find /opt/CA/ -type d -exec sh -c 'ls -1 "{}"|wc -l' \; -print | sed '$!N;s/\n/ /' | sort -n

        From the output, remove files from the directory that has the most number of files. Deleting files result in freeing up their associate inodes.

Once you have increased the free inodes, restart the CA WA System Agent and check if the jobs are running successfully.

Additional Information

 

The CA WA System Agent spool logs may over a period of time accumulate and use up inodes. It is important to ensure the spool directory is maintained (housekeeping) periodically.