Memory allocation errors - MergeTree background tasks failing in Clickhouse Keeper instance for Watchtower

Products

• WatchTower
• WatchTower Platform

Issue/Introduction

We recently started getting rolling mergeTree background tasks failing in our Keeper instance due to the tasks trying to allocate too much memory to the task. We've bumped up the memory for the pods and are still getting the same issue. This is causing upstream impact to the DBLoader instance, but hasn't caused further upstream impact at this time.

Environment

Watchtower 1.3

Cause

In looking at the Watchtower Diagnostics Dump, can see that this is a limitation from the current Clickhouse Keeper where it internally hosts a Clickhouse DB that takes up the resources.  This will be addressed by Clickhouse for WT 1.4 release.  For WT 1.3, in validating the steps to resolve this, will share  instructions to resolve.

Resolution

Provided an Instructions PDF file titled "How to Disable the Internal Database in Clickhouse Keeper [WT 1.3.x]" to resolve, attached to the KD article.