The Redis Service crashing due to almost 100% memory utilization
search cancel

The Redis Service crashing due to almost 100% memory utilization

book

Article ID: 376617

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • Redis Service crashing due to nearly 100% Memory Utilization.
  • RaaS server is logging messages related to command compilation.
    2024-08-23 14:22:32,311 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-1:4385] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 823387}
    2024-08-23 14:22:32,311 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-7:4421] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 1083835}
    2024-08-23 14:22:32,311 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-3:4397] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 807757}
    2024-08-23 14:22:32,312 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-6:4415] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 444782}

Environment

Aria Automation Config 8.x
(Tanzu Salt)

Cause

  • The Redis server is using nearly 100% of its memory.
  • The Redis database has grown to 17 GB, and the Celery queue depth is increasing exponentially.

Resolution

This needs to be addressed at the database level. Please submit a support case by following the instructions in the KB: KB142884.