API execution is failing while trying to delete Clickhouse Backup

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

API execution is failing while trying to delete Clickhouse Backup due to which VCO store3 directory was consuming huge amount of space since clickhouse backup files were not deleted from /store3/clickhouse/shadow/

root@LKT-CTF-VCO1:/store3/clickhouse/shadow# df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                      63G     0   63G   0% /dev
tmpfs                     13G   17M   13G   1% /run
/dev/mapper/vols-root1    12G  6.5G  4.7G  59% /
tmpfs                     63G     0   63G   0% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                     63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/store2-data  394G   73G  306G  20% /store2
/dev/mapper/store3-data  3.1T  2.6T  454G  85% /store3
/dev/mapper/store-data   3.4T  384G  2.9T  12% /store
/dev/mapper/vols-var1    3.9G  2.5G  1.2G  69% /var
/dev/mapper/vols-vartmp  2.0G   48K  1.8G   1% /var/tmp
/dev/mapper/vols-home    2.0G  170M  1.7G  10% /home
/dev/mapper/vols-log      84G   59G   22G  74% /var/log
/dev/mapper/vols-tmp     2.0G   60K  1.8G   1% /tmp
/dev/mapper/vols-audit   2.0G  736M  1.1G  41% /var/log/audit
tmpfs                     13G     0   13G   0% /run/user/998
overlay                  394G   73G  306G  20% /store2/docker/overlay2/3fb141eba791f1df2ddcb1fe666527c5aa5011d8af31a8b0728c1d8f8130726a/merged
overlay                  394G   73G  306G  20% /store2/docker/overlay2/d8c576865d0080aeb1b4993db947c558d9066f084e0d837ab10ce44f20ae5c1c/merged
overlay                  394G   73G  306G  20% /store2/docker/overlay2/30afcf7ffc6843842e989d618137615bcedad89039ea1fe12564851cd5e1b987/merged
overlay                  394G   73G  306G  20% /store2/docker/overlay2/e29b560e9ad7119147b6c93103e0605927ea2a92c2a04941fa062c24fa4f7ca6/merged
overlay                  394G   73G  306G  20% /store2/docker/overlay2/a0b0f3ee625970a64ec0f50da6d27f18ca4daf66db2ec54a651ca1162420ab09/merged
overlay                  394G   73G  306G  20% /store2/docker/overlay2/ae4e784f3132c5cb8159f9ba4aa2471bd03b39d9624388bcd1aa1ff032184107/merged
overlay                  394G   73G  306G  20% /store2/docker/overlay2/8e2c5e376da3a94b70055a143ecc7304f8016fd4aa204a8c2f888cd7bfdf1826/merged
tmpfs                     13G     0   13G   0% /run/user/1001

 

Clickhouse backup file was taking lot of space. 

root@LKT-CTF-VCO1:/store3/clickhouse/shadow# ls -l
total 8
drwxr-x--- 3 clickhouse clickhouse 4096 Nov 20 17:11 1700500260007_backup
-rw-r----- 1 clickhouse clickhouse    8 Apr 30 06:41 increment.txt
root@LKT-CTF-VCO1:/store3/clickhouse/shadow#

Environment

VMware SDWAN Orchestrator

Cause

API  /disasterRecovery/deleteClickhouseBackup is responsible to delete the clickhouse backups on Active VCO once it is transferred to Standby VCO.

This API failed to execute due to which the clickhouse backup was not deleted.

 

2023-11-20T17:12:01.974Z - error: [process.copy-fil~sync.170006002.16552] [28067] Error calling /disasterRecovery/deleteClickhouseBackup on remote vco Error: Cannot enqueue Query after fatal error.
    at deleteClickhouseBackupOnActive (/usr/share/node/node_modules/velocloud/lib/disasterRecovery/drutils.js:1223:15)

Resolution

We have to delete the *_backup file manually from /store3/clickhouse/shadow/ directory

root@LKT-CTF-VCO1:/store3/clickhouse/shadow# ls -l
total 8
drwxr-x--- 3 clickhouse clickhouse 4096 Nov 20 17:11 1700500260007_backup
-rw-r----- 1 clickhouse clickhouse    8 Apr 30 06:41 increment.txt
root@LKT-CTF-VCO1:/store3/clickhouse/shadow#

NOTE: We need to delete only the *_backup folder. Not the shadow folder or increment.txt.