“Error received from agent: ERRORED” when running a data collection on CCS agents.
Article ID: 223150
Updated On:
Products
Issue/Introduction
Control Compliance Suite (CCS)
When running a data collection scan on a Linux agent with a large File System, the following error would sometimes be reported:
“Error received from agent: ERRORED”
Also 'BvCUUNIXDataStoreWrapper.cp' entries can be found in the /esm/system/<hostname>/DCModule.log file on the agent, and this error shows that the job is failing because the system is running out of disk space:
Example of errors that are found in the DCModule.log:
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 512|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 500|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 195|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 441|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 512|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 500|Error: - 0x1c
35FE|2021/08/15|19:53:15|BvCUUNIXDataStoreWrapper.cp| 195|Error: - 0x1c
Environment
Release : CCS 12.5.x
Component : CCS Unix\Linux agent
Cause
The agent ran out of disk space to store the temporary files for the CCS scan.
Typically agents reporting this error are running checks that would report on how many world-writable and unowned files that are on the system. On servers that have shared drives with millions of files, just one of these types of checks was collecting information on all unowned files, and producing temporary file on the agent which equaled over 7 GB in size (average 1k bytes temp disk space per unowned file discovered by the scan). Since the server only had 3 GB free space available, it would run out space and this error was reported. See the 'Additional Information' section below for more information.
Resolution
Can be resolved a couple of ways:
- Determine which custom checks that are causing the problem (run them one at a time until to narrow it down) and then remove that check(s) from being run on those agent(s).
- Increase the free space on the agent to accommodate the amount of temporary space that will be needed to be able to run the selected checks.
- Customize the checks to exclude specific directories that contain a large amount of files.
- KB on technical support on custom checks and standards: Technical support for CCS Standards and Checks
Additional information on agents with large file systems:
Typically the checks that can cause an agent that has a large File System to run out of disk space are checks that report the 'number of world writable files', 'number of world writable directories without sticky bit', and/or the 'number of unowned files or directories' on the server. To get an idea on how many files of those types are on the server, you can run the following commands outside of CCS to see how many of those specific types are on the agent server.
- Command #1 = Display the number of world writable files:
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002 2>/dev/null | wc -l
- Command #2 = Display the number of world-writable directories without sticky bit:
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type d \( -perm -0002 -a ! -perm -1000 \) 2>/dev/null | wc -l
- Command #3 = Display the number of unowned files or directories:
df --local -P | awk {'if (NR!=1) print $6'} | xargs -I '{}' find '{}' -xdev -nouser 2>/dev/null | wc -l
When CCS runs a check, this activity creates a 'response' .xml file in temporary location on the agent server for each file/directory that is found. Each response .xml record requires 360 + full filename length bytes (longer the path, larger the file record size). Assuming that on average 1KB temporary disk space is used per response file; it will require <results from one of the commands above> * 1K bytes to get an idea of how much free space on the agent server will need to run the check.
For example, if you run the command #3 above to display the number of unowned files or directories, and it returns 5288990 results, then take 5288990 * 1K bytes and that is around 5.2 gigs of temporary disk space that the one check alone will need on the agent for the check to run correctly.
Additional Information
In one case the “Error received from agent: ERRORED” message appeared when running one of the Predefined CIS Benchmark standards on some Linux servers. It looked like the Predefined standard somehow became corrupted and was causing this error. Once they deleted and added that predefined standard again (they were at the newest SCU at the time) fixed the issue and they were able to run the CIS benchmark without any errors.
Feedback
