/storage/dblog partition usage rate on vCenter increased sharply
search cancel

/storage/dblog partition usage rate on vCenter increased sharply

book

Article ID: 312441

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • /storage/dblog partition in vCenter shows a usage rate of 65% and above occasionally. However, this may not be showing any impact on the environment. 
  • Below is the scenario -
    root@vc_name [ ~ ]# df -h
    Filesystem                                   Size  Used Avail Use% Mounted on
    devtmpfs                                     4.0M     0  4.0M   0% /dev
    tmpfs                                        6.9G  952K  6.9G   1% /dev/shm
    tmpfs                                        2.8G  1.3M  2.8G   1% /run
    tmpfs                                        4.0M     0  4.0M   0% /sys/fs/cgroup
    /dev/mapper/vg_root_0-lv_root_0               47G   13G   32G  29% /
    /dev/sda3                                    488M   37M  415M   9% /boot
    /dev/sda2                                     10M  2.0M  8.1M  20% /boot/efi
    tmpfs                                        6.9G  4.9M  6.9G   1% /tmp
    /dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot  492G   28K  467G   1% /storage/lvm_snapshot
    /dev/mapper/lifecycle_vg-lifecycle            98G  3.9G   90G   5% /storage/lifecycle
    /dev/mapper/imagebuilder_vg-imagebuilder     9.8G   36K  9.3G   1% /storage/imagebuilder
    /dev/mapper/netdump_vg-netdump               982M   24K  915M   1% /storage/netdump
    /dev/mapper/autodeploy_vg-autodeploy         9.8G   40K  9.3G   1% /storage/autodeploy
    /dev/mapper/core_vg-core                      25G  1.8M   24G   1% /storage/core
    /dev/mapper/vtsdblog_vg-vtsdblog              15G   33M   14G   1% /storage/vtsdblog
    /dev/mapper/updatemgr_vg-updatemgr            98G  569M   93G   1% /storage/updatemgr
    /dev/mapper/log_vg-log                       9.8G  1.9G  7.4G  21% /storage/log
    /dev/mapper/vtsdb_vg-vtsdb                   541G   35M  513G   1% /storage/vtsdb
    /dev/mapper/db_vg-db                         9.8G  291M  9.0G   4% /storage/db
    /dev/mapper/dblog_vg-dblog                    15G  9.4G  4.6G  68% /storage/dblog
    /dev/mapper/archive_vg-archive                49G  7.1G   40G  16% /storage/archive
    /dev/mapper/seat_vg-seat                     541G  174M  513G   1% /storage/seat

    du -a /storage/dblog/ | sort -n -r | head -n 20
    total 16G
    drwx------ 2 vpostgres vpgmongrp 4.0K May  3  2023  [0m [01;34marchive_status [0m
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000ED
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000EA
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000EE
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:40 00000001000002BF000000F0
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:44 00000001000002BF000000E9
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:50 00000001000002BF000000EC
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:55 00000001000002BF000000EB
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:55 00000001000002BF000000EF
    -rw------- 1 vpostgres vpgmongrp  16M Mar 18 19:57 00000001000002BF000000F4


    root@vc_name[ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -c "select name, setting, unit from pg_settings where name in ('min_wal_size', 'max_wal_size')"
         name | setting | unit
    --------------+---------+------
     max_wal_size | 655 | 16MB
     min_wal_size | 655 | 16MB
    (2 rows)
  • From above parameter, 16M*655= 10480 MB = 10.5GB < 15G
    root@vc_name [ ~ ]# /opt/vmware/vpostgres/current/bin/psql -U postgres -c "select pg_postmaster_start_time()"
       pg_postmaster_start_time
    -------------------------------
     yyyy-mm-dd22:12:07.604504+00
    (1 row)

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Resolution

  • The data from /storage/dblog/vpostgres/pg_xlog is needed for the Postgres to recover in case of a crash. WAL logs are stored in the directory > pg_xlog under the data directory, as a set of segment files, normally each 16 MB in size.
  • The /Storage/dblog can hit up to 80%, which is basically by design to safeguard the postgres database. There are safety mechanisms in vpostgres to make sure that even if vmware-vpostgres-archiver is stopped or isn't working for any reason the dblog usage won't reach 100%.
  • Do not delete /storage/dblog/vpostgres/pg_xlog manually, as these files contain the postgres process and deleting which could impact the database functionality and may lead to a situation where we have to restore vCenter from a healthy backup or reinstall it, if no backup is available.
  • The size of the /storage/dblog partition can be increased if needed, but if still the usage crosses the threshold of 80%, this could be an indication that there is some issue with the database which needs further troubleshooting.
  • The recommendation is to gather the vCenter logs and create a service request for further investigation by Broadcom support assistance.
Powered by Wolken Software