How to solve /store3 and /var/log disk full issue
Article ID: 373461
Updated On:
Products
Issue/Introduction
This KB is related to VCO disk full issue.
For on-premise customers, the disk will be full at some case, which may cause problems with VCO operation. This KB describes how to solve the each pattern of disk full.
At first, please get "df -h" to see which folder is full. If the USE% is100%, that disk is full.
For example.
df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 12M 6.3G 1% /run
/dev/mapper/vols-root1 20G 5.5G 14G 30% /
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/store-data 807G 146G 629G 19% /store
/dev/mapper/store2-data 394G 147G 232G 39% /store2
/dev/mapper/store3-data 807G 774G 0 100% /store3 <-------------------disk full
/dev/mapper/vols-home 2.0G 164K 1.8G 1% /home
/dev/mapper/vols-var1 7.8G 210M 7.3G 3% /var
/dev/mapper/vols-tmp 2.0G 68K 1.8G 1% /tmp
/dev/mapper/vols-vartmp 3.9G 48K 3.7G 1% /var/tmp
/dev/mapper/vols-log 50G 47G 0 100% /var/log <-------------------disk full
/dev/mapper/vols-audit 2.0G 1.4G 489M 74% /var/log/audit
tmpfs 6.3G 0 6.3G 0% /run/user/1001
Resolution
How to solve with each pattern is the following:
Pattern 1. /store3 disk size full
The cause of this issue is clickhouse log increasing.
Please get "du -h /store3/clickhouse/" on VCO to check which clickhouse logs are overwhelming the capacity.
Workaround:
Basically, you can expand Disk Size to fix this issue. For the procedure, please refer to the following documents.
https://docs.vmware.com/en/VMware-SD-WAN/6.0/sd-wan-orchestrator-deployment-and-monitoring-guide/GUID-A1ED5E3C-2B5D-4719-9D5B-143BB3F47FE2.html
However, if you can not expand Disk Size soon because not prepare additional physical disk, you can control store3 utilization if you configure the stats related retention system properties.
For procedure of chanfing the stats related retention system properties, please refer to the following documents.
https://docs.vmware.com/en/VMware-SD-WAN/5.0/VMware-SD-WAN-Administration-Guide/GUID-E4C1371D-0E34-44FB-9BF6-835F313C58C2.html
If the backup folder (whome name ends with “_backup”) in the "/store3/clickhouse/shadow" folder is too large, you can also solve the problem by deleting the backup folder.
NOTE: Please delete only the *_backup folder. not the shadow folder or increment.txt.
Pattern 2. /var/log size full
If ownership is incorrect, log rotations may fail and capacity may be exhausted.
Please get "sudo ls -lR /var/log/" to check the ownership of log files and which file is huge size. After that please inquery technical support.
Feedback
