Post upgrade to 1371, sensor's Disk is filling up with high disk usage alert due to rapid growing log

Products

• VMware vDefend Network Detection and Response

Issue/Introduction

High disk usage alert in the sensor due to rapid growing of the of log

Environment

In the on-prem Sensor in 1371 version

Cause

Disk would start filling up with logs

1. Check the usage on the root partition:

lastline-df -d -h

Sample:

Filesystem                             Size  Used Avail Use% Mounted on
udev                                    16G     0   16G   0% /dev
tmpfs                                  3.1G  5.2M  3.1G   1% /run
/dev/mapper/lastline--sensor--vg-root  914G  868G  297M 100% /
tmpfs                                   16G   24K   16G   1% /dev/shm
tmpfs                                  5.0M     0  5.0M   0% /run/lock
tmpfs                                   16G     0   16G   0% /sys/fs/cgroup
/dev/sda2                              513M  4.0K  513M   1% /boot/efi
tmpfs                                  3.1G     0  3.1G   0% /run/user/1000

2. Check the folders taking the most disk space:
du -xah --time --max-depth=3 /var | sort | grep G

Sample:

300G   2023-03-06 14:55   /var/log

 

Disk would start filling up with logs from /var/log/suricata-eve/surcata-eve-fileinfo* 

This can be observed with frequent logs entries like below in /var/log/suricata-eve/suricata-eve-fileinfo.error.log:

Aug 20 06:42:33 lastline-sensor suricata-eve_suricata-eve-fileinfo_2[2500]: suricata_eve.workers.fileinfo - ERROR - Error processing event {'timestamp': '2024-08-20T06:42:33.590409+0000', 'src_ip': 'X.X.X.X', 'src_port': 443, 'dest_ip': 'X.X.X.X', 'dest_port': 52876, 'proto': 'TCP', 'direction': 'to_client', 'flow_id': 683925136486077, 'in_iface': 'ens1f1', 'ether': {'src_mac': '00:1c:7f:a4:b2:33', 'dest_mac': 'b4:0c:25:e0:c0:44'}, 'app_proto': 'http2', 'event_type': 'fileinfo', 'fileinfo': {'sid': [], 'gaps': False, 'state': 'CLOSED', 'tx_id': 127, 'sha256': 'e2a6d14997ecccef0e7307ed9d6b426ea8362dc786d1e9f33a65b54ba7a85cfd', 'stored': False, 'size': 429}}: 'http2'

This results into a huge log files leading to High disk usage warnings/alert.

This is due to the bug in 1371 : no support to http2 

 

Resolution

workaround:

1. Run the command on the sensor as root: sed -i -e "s/app_layer:/app_layer:\n http2:\n enabled: false/g" /usr/share/appliance-config/modules/timon/templates/tiller_config.yaml.erb
2. Then re-trigger the configuration.

Additional Information

Fix available in 9.8.1 release