API execution is failing while trying to delete Clickhouse Backup due to which VCO store3 directory was consuming huge amount of space since clickhouse backup files were not deleted from /store3/clickhouse/shadow/
root@LKT-CTF-VCO1:/store3/clickhouse/shadow# df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 17M 13G 1% /run
/dev/mapper/vols-root1 12G 6.5G 4.7G 59% /
tmpfs 63G 0 63G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/mapper/store2-data 394G 73G 306G 20% /store2
/dev/mapper/store3-data 3.1T 2.6T 454G 85% /store3
/dev/mapper/store-data 3.4T 384G 2.9T 12% /store
/dev/mapper/vols-var1 3.9G 2.5G 1.2G 69% /var
/dev/mapper/vols-vartmp 2.0G 48K 1.8G 1% /var/tmp
/dev/mapper/vols-home 2.0G 170M 1.7G 10% /home
/dev/mapper/vols-log 84G 59G 22G 74% /var/log
/dev/mapper/vols-tmp 2.0G 60K 1.8G 1% /tmp
/dev/mapper/vols-audit 2.0G 736M 1.1G 41% /var/log/audit
tmpfs 13G 0 13G 0% /run/user/998
overlay 394G 73G 306G 20% /store2/docker/overlay2/3fb141eba791f1df2ddcb1fe666527c5aa5011d8af31a8b0728c1d8f8130726a/merged
overlay 394G 73G 306G 20% /store2/docker/overlay2/d8c576865d0080aeb1b4993db947c558d9066f084e0d837ab10ce44f20ae5c1c/merged
overlay 394G 73G 306G 20% /store2/docker/overlay2/30afcf7ffc6843842e989d618137615bcedad89039ea1fe12564851cd5e1b987/merged
overlay 394G 73G 306G 20% /store2/docker/overlay2/e29b560e9ad7119147b6c93103e0605927ea2a92c2a04941fa062c24fa4f7ca6/merged
overlay 394G 73G 306G 20% /store2/docker/overlay2/a0b0f3ee625970a64ec0f50da6d27f18ca4daf66db2ec54a651ca1162420ab09/merged
overlay 394G 73G 306G 20% /store2/docker/overlay2/ae4e784f3132c5cb8159f9ba4aa2471bd03b39d9624388bcd1aa1ff032184107/merged
overlay 394G 73G 306G 20% /store2/docker/overlay2/8e2c5e376da3a94b70055a143ecc7304f8016fd4aa204a8c2f888cd7bfdf1826/merged
tmpfs 13G 0 13G 0% /run/user/1001
Clickhouse backup file was taking lot of space.
root@LKT-CTF-VCO1:/store3/clickhouse/shadow# ls -l
total 8
drwxr-x--- 3 clickhouse clickhouse 4096 Nov 20 17:11 1700500260007_backup
-rw-r----- 1 clickhouse clickhouse 8 Apr 30 06:41 increment.txt
root@LKT-CTF-VCO1:/store3/clickhouse/shadow#
VMware SDWAN Orchestrator
API /disasterRecovery/deleteClickhouseBackup is responsible to delete the clickhouse backups on Active VCO once it is transferred to Standby VCO.
This API failed to execute due to which the clickhouse backup was not deleted.
2023-11-20T17:12:01.974Z - error: [process.copy-fil~sync.170006002.16552] [28067] Error calling /disasterRecovery/deleteClickhouseBackup on remote vco Error: Cannot enqueue Query after fatal error.
at deleteClickhouseBackupOnActive (/usr/share/node/node_modules/velocloud/lib/disasterRecovery/drutils.js:1223:15)
We have to delete the *_backup file manually from /store3/clickhouse/shadow/ directory
root@LKT-CTF-VCO1:/store3/clickhouse/shadow# ls -l
total 8
drwxr-x--- 3 clickhouse clickhouse 4096 Nov 20 17:11 1700500260007_backup
-rw-r----- 1 clickhouse clickhouse 8 Apr 30 06:41 increment.txt
root@LKT-CTF-VCO1:/store3/clickhouse/shadow#
NOTE: We need to delete only the *_backup folder. Not the shadow folder or increment.txt.