Error: No healthy upstream due to /storage/db partition full on vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server UI is inaccessible with error: No healthy upstream.

• vCenter Server log files contain the following errors:

  • /var/log/vmware/vpxd/vpxd.log: No Space left on the device.

  • /storage/db/vpostgres/pg_log/postgresql.log: psql.bin: could not extend file "pg_tblspc/####/PG_9.4_####/####/####": No space left on device.

• Command df -h confirms /storage/db partition is full.

Environment

VMware vCenter Server 6.x

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

Accumulation of redundant data causes database bloat, filling up the /storage/db partition. Additional contributing factors include:

• Known bugs related to database.

• Persistent idle transactions.

• Ineffective internal database maintenance processes.

• Failure of the vCenter Server operating system to manage the automatic database cleanup utility threads.

Resolution

Note: Prior to executing the following steps, take a backup or an offline snapshot (powered-off state) of the vCenter Server. If the vCenter Server is part of an Enhanced Linked Mode (ELM) setup, simultaneous snapshots must be taken for all replicating vCenter Servers. Refer VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 

To remediate database bloat and reclaim disk space, perform the following steps:

1. Reboot vCenter Server: Rebooting restores lost process threads for automatic cleanup.

   1. Run df -h and record current /storage/db utilization.

   2. Identify the ESXi host managing the vCenter Server.

   3. Log in to the ESXi host Client at https://<ESXi_IP>.

   4. Navigate to the vCenter Server VM > Shut Down Guest OS. 

      1. Wait 30 minutes after the VCSA reboots.

   5. Run df -h. If the utilization remains unchanged, proceed to section 2.

2. Extend Partition (If the partition is 100% full): The database cleanup utility requires overhead space to function.

   1. Increase the virtual disk size and extend the partition.

   2. Refer to Increasing the disk space for the VMware vCenter Server Appliance for instructions.

3. Analyze database bloat:

   1. Download the check_bloat.sql script attached to this article.

   2. Execute the script: psql -d VCDB -U postgres -f check_bloat.sql

   3. Review the tbloat and ibloat columns. If any value exceeds 25, proceed to remove bloat.

   4. To identify specific high-utilization tables, run:

      SELECT nspname || '.' || relname AS "relation",
      pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
      FROM pg_class C
      LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
      WHERE nspname NOT IN ('pg_catalog', 'information_schema')
      AND C.relkind <> 'i'
      AND nspname !~ '^pg_toast'
      ORDER BY pg_total_relation_size(C.oid) DESC
      LIMIT 20;

4. Remove database bloat :

   1. Stop all vCenter Server services: service-control --stop --all

   2. Start the vPostgres service: service-control --start vmware-vpostgres

   3. Connect to the database: /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB

   4. Rebuild all tables and indexes: vacuum full;

   5. Update statistics for the query planner: vacuum analyze;

   6. Exit the prompt: \q

   7. Start all vCenter Server services: service-control --start --all

Additional Information

vCenter Server Appliance Data Integrity Best Practices