Add option for the Filebeat and Logcollector images to log the output into STDOUT without logging into files

Products

• DX Operational Intelligence

Issue/Introduction

Creating enhancement request for the required change in the base images of Filebeat and Logcollector so that no additional customization is required by the users of these images:

In order to use Filebeat and Logcollector images without any modifications, we need to have the option to log Filebeat and Logcollector output into STDOUT without logging into files. Currently we have custom images with startup scripts which disabled logging into log files, but in this case we need to re-create images each time FB or LC image is changed. Moreover, in case FB or LC startup script is changed, we need to merge them with our changes. 
It is possible to use logging into files with minimal log file sizes, but in this case we need to add additional ephemeral storage to pods just for logs we don't need. We want to avoid any unneeded resource allocation.

Requested changes:

1. Filebeat
We already have a custom filebeat.yml file, and can redirect logging into STDERR stream using logging.to_stderr:true . All that we need from AIOPS team is redirection of STDERR into STDOUT on Filebeat startup by replacing
./filebeat &

with 

./filebeat 2>&1 &

in startFilebeat.sh file.

2. Logcollector:
To turn of logging into the files, we need to remove (or comment) following lines in log4j2.properties file:
rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling

logger.slowlog.appenderRef.rolling_slowlog.ref = ${sys:ls.log.format}_rolling_slowlog
We have code for this purpose in our startup script:
sed -i "s|rootLogger.appenderRef.rolling|#rootLogger.appenderRef.rolling|g" $LOGSTASH_LOG4J_CONFIG_FILE
sed -i "s|logger.slowlog.appenderRef.rolling_slowlog|#logger.slowlog.appenderRef.rolling_slowlog|g" $LOGSTASH_LOG4J_CONFIG_FILE

Also, we need to remove redirection of  STDOUT into /dev/null in startLogCollector.sh by replacing
bin/logstash -f $LOGCOLLECTOR_CONFIG_DIR/conf --config.reload.automatic -$LOGCOLLECTOR_WORKER_THREADS -$LOGCOLLECTOR_BATCH_SIZE --pipeline.batch.delay $LOGCOLLECTOR_BATCH_DELAY &> /dev/null

with
bin/logstash -f $LOGCOLLECTOR_CONFIG_DIR/conf --config.reload.automatic -$LOGCOLLECTOR_WORKER_THREADS -$LOGCOLLECTOR_BATCH_SIZE --pipeline.batch.delay $LOGCOLLECTOR_BATCH_DELAY 2>&1 &

This can be controlled by some environment variable (LOGCOLLECTOR_LOGGER_USE_STDOUT?)

Environment

Release : 21.3

Component : CA DOI ALARM ANALYTICS

Resolution

An enhancement requested was already raised in order to Add option for the Filebeat and Logcollector images to log the output into STDOUT without logging into files, it will be evaluated and put in our backlog to be deployed soon.

To follow up on the status of this request, please raise a ticket on https://support.broadcom.com/ and refer to this knowledge article in order to get the status.