/storage/dblog usage rate increased sharply
search cancel

/storage/dblog usage rate increased sharply

book

Article ID: 312441

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms -

  • Customers may come up with the concern that the /storage/dblog shows as a usage rate of 65% and above occasionally. This may not be showing any impact on the environment. 
  • Below is the scenario -
root@vc_name [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 0 7.9G 0% /dev
tmpfs 7.9G 740K 7.9G 1% /dev/shm
tmpfs 7.9G 684K 7.9G 1% /run
tmpfs 7.9G 0 7.9G 0% /sys/fs/cgroup
/dev/sda3 11G 5.9G 4.3G 59% /
tmpfs 7.9G 1.6M 7.9G 1% /tmp
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/db_vg-db 9.8G 364M 8.9G 4% /storage/db
/dev/mapper/dblog_vg-dblog 15G 9.4G 4.6G 68% /storage/dblog 
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/log_vg-log 9.8G 2.9G 6.4G 31% /storage/log
/dev/mapper/autodeploy_vg-autodeploy 9.8G 23M 9.2G 1% /storage/autodeploy
/dev/mapper/archive_vg-archive 50G 14G 34G 29% /storage/archive
/dev/sda1 120M 34M 77M 31% /boot
/dev/mapper/updatemgr_vg-updatemgr 99G 79M 94G 1% /storage/updatemgr
/dev/mapper/seat_vg-seat 25G 1.4G 22G 6% /storage/seat
/dev/mapper/core_vg-core 50G 2.3G 45G 5% /storage/core


du -a /storage/dblog/ | sort -n -r | head -n 20
total 16G
drwx------ 2 vpostgres vpgmongrp 4.0K May  3  2023  [0m [01;34marchive_status [0m
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000ED
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000EA
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000EE
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000F0
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:44 00000001000002BF000000E9
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:50 00000001000002BF000000EC
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:55 00000001000002BF000000EB
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:55 00000001000002BF000000EF
-rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:57 00000001000002BF000000F4


root@vc_name[ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -c "select name, setting, unit from pg_settings where name in ('min_wal_size', 'max_wal_size')"
     name | setting | unit
--------------+---------+------
 max_wal_size | 655 | 16MB
 min_wal_size | 655 | 16MB
(2 rows)
  • From above parameter, 16M*655= 10480 MB = 10.5GB < 15G
root@vc_name [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -c "select pg_postmaster_start_time()"
   pg_postmaster_start_time
-------------------------------
 yyyy-mm-dd22:12:07.604504+00
(1 row)

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Resolution

  • The data from /storage/dblog/vpostgres/pg_xlog is needed for the Postgres to recover in case of a crash. WAL logs are stored in the directory > pg_xlog under the data directory, as a set of segment files, normally each 16 MB in size.
  • The /Storage/dblog can hit up to 80%, which is basically by design to safeguard the postgres database. There are safety mechanisms in vpostgres to make sure that even if vmware-vpostgres-archiver is stopped or isn't working for any reason the dblog usage won't reach 100%.
  • Do not delete /storage/dblog/vpostgres/pg_xlog manually, as these files contain the postgres process and deleting which could impact the database functionality and customers will have to restore the backup or re-install as a last resort.
  • Customers can increase the size of the /storage/dblog partition if needed, but if still the usage crosses the threshold of 80%, this could be an indication that there is some issue with the database which would need to troubleshoot. The recommendation is to gather the required logs and information and create a PR for the same.