ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

“Error received from agent: ERRORED” when running a data collection on CCS agents.

book

Article ID: 223150

calendar_today

Updated On:

Products

Control Compliance Suite Standards Database Control Compliance Suite Control Compliance Suite Standards Server

Issue/Introduction

Control Compliance Suite (CCS)

When running a data collection scan on a Linux agent with a large File System, the following error would sometimes be reported:

“Error received from agent: ERRORED”

 

Also 'BvCUUNIXDataStoreWrapper.cp' entries can be found in the DCModule.log file on the agent, and this error shows that the job is failing because the system is running out of disk space:

Example of errors that are found in the DCModule.log:

35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 512|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 500|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 195|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 441|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 512|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 500|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 195|Error: - 0x1c

Cause

The agent ran out of disk space to store the temporary files for the CCS scan.

The agent was running a couple custom checks in a Standard that would report on how many world-writable and unowned files were on the system.  Since this server had shared drives with millions of files, just one check was collecting information on all unowned files and producing temporary file on the agent which equaled over 7 GB in size (average 1k bytes temp disk space per unowned file discovered by the scan), and the agent was running out of free disk space (it had less than 3 GB free space available).

Environment

Release : CCS 12.5.x

Component : CCS Unix\Linux agent

Resolution

Can be resolved a couple of ways:

  • Determine which custom checks that are causing the problem (run them one at a time until to narrow it down) and then remove those checks from being run on those agent(s).
  • Increase the free space on the agent to accommodate the about of temporary space that will be needed to be able to run the selected checks.
  • Customize the checks to exclude specific directories that contain a large amount of files.

 

Additional Information

Additional information on agents with large file systems:

Typically the checks that can cause an agent that has a large File System to run out of disk space are checks that report the 'number of world writable files', 'number of world writable directories without sticky bit', and/or the 'number of unowned files or directories' on the server.  To get an idea on how many files of those types are on the server, you can run the following commands outside of CCS to see how many of those specific types are on the agent server.  

  • Command #1 = Display the number of world writable files:
    df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002 2>/dev/null | wc -l

 

  • Command #2 = Display the number of world-writable directories without sticky bit:
    df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type d \( -perm -0002 -a ! -perm -1000 \) 2>/dev/null | wc -l

 

  • Command #3 = Display the number of unowned files or directories:
    df --local -P | awk {'if (NR!=1) print $6'} | xargs -I '{}' find '{}' -xdev -nouser 2>/dev/null | wc -l

 

When CCS runs a check, this activity creates a 'response' .xml file in temporary location on the agent server for each file/directory that is found. Each response .xml record requires 360 + full filename length bytes (longer the path, larger the file record size). Assuming that on average 1KB temporary disk space is used per response file; it will require <results from one of the commands above> * 1K bytes to get an idea of how much free space on the agent server will need to run the check.

For example, if you run the command #3 above to display the number of unowned files or directories, and it returns 5288990 results, then take 5288990 * 1K bytes and that is around 5.2 gigs of temporary disk space that the one check alone will need on the agent for the check to run correctly.