The Redis Service crashing due to almost 100% memory utilization

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Redis Service crashing due to nearly 100% Memory Utilization.
• RaaS server is logging messages related to command compilation.
  2024-08-23 14:22:32,311 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-1:4385] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 823387}
2024-08-23 14:22:32,311 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-7:4421] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 1083835}
2024-08-23 14:22:32,311 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-3:4397] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 807757}
2024-08-23 14:22:32,312 [var.lib.raas.unpack._MEIWmzae7.raas.mods.db.schedule              ][WARNING :216 ][ForkPoolWorker-6:4415] Compile commands warn threshold (1000) exceeded. Stats = {'cmd_compile': 444782}

Environment

Aria Automation Config 8.x
(Tanzu Salt)

Cause

• The Redis server is using nearly 100% of its memory.
• The Redis database has grown to 17 GB, and the Celery queue depth is increasing exponentially.
  
• 

Resolution

This needs to be addressed at the database level. Please submit a support case by following the instructions in the KB: KB142884.