Apply Changes fails with clamd error

Products

• VMware Tanzu Gemfire

Issue/Introduction

How to fix clamd issue when Apply Changes fails on VMware Tanzu Gemfire.

The Apply Changes sometimes dies shortly after the Service VM for VMware Tanzu Gemfire is spun up and throws the following error:

Task 88492 | 21:20:05 | Updating instance server: server/68d991bd-2537-4054-8cf7-2f7c415f73c6 (0) (canary) (00:11:57)
                      L Error: 'server/68d991bd-2537-4054-8cf7-2f7c415f73c6 (0)' is not running after update. Review logs for failed jobs: clamd
Task 88492 | 21:32:02 | Error: 'server/68d991bd-2537-4054-8cf7-2f7c415f73c6 (0)' is not running after update. Review logs for failed jobs: clamd

Task 88492 Started  Mon Jan 28 21:06:54 UTC 2019
Task 88492 Finished Mon Jan 28 21:32:02 UTC 2019
Task 88492 Duration 00:25:08
Task 88492 error

FreshClam.log (/var/vcap/sys/log) shows the following error:

ClamAV update process started at Wed Jan 30 20:28:43 2019
Reading CVD header (main.cld): WARNING: main.cld not found on remote server
Reading CVD header (main.cvd): OK
main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Reading CVD header (daily.cld): WARNING: daily.cld not found on remote server
Reading CVD header (daily.cvd): OK
daily.cvd is up to date (version: 25343, sigs: 2228905, f-level: 63, builder: raynman)
Reading CVD header (bytecode.cld): WARNING: bytecode.cld not found on remote server
Reading CVD header (bytecode.cvd): OK
bytecode.cvd is up to date (version: 328, sigs: 94, f-level: 63, builder: neo)

Note: clamd only fails on servers and not on the locators.

Environment

Product Version: 1.5

Resolution

The issue is due to the default memory cgroup of clamav being too low to load the clamav virus definitions into memory and OS swap being off in VMware Tanzu Gemfire. 

This issue can be fixed by updating the clamav runtime config to include the following property: 

addons:
- name: clamav
  jobs:
  - name: clamav
    release: clamav
  properties:
    clamav:
      memory_limit: 1500000000

By default, ClamAV (until version 1.4.39), consumes 1 GB of memory. As a result, increasing the cgroup memory limit to 1.5 GB (1500000000) will resolve this issue.