Reclaiming space on vPostgres databases within the VMware vCenter Server Appliance fails with the error: domain socket "/tmp/.s.PGSQL.5432"

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Reclaiming space on vPostgres database with in the VMware vCenter Server Appliance fails
• When you run the vacuumdb script # sudo -u postgres /opt/vmware/vpostgres/1.0/bin/vacuumdb -a -e -v -f -U postgres > /tmp/vacuumdb.log, you see the error:
  
  vacuumdb: could not connect to database postgres: could not connect to server: No such file or directory
  Is the server running locally and accepting connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
  
  
• In the /storage/db/vpostgres/serverlog file, you see entries similar to:
  
  <YYYY-MM-DD>T<time> UTC db: pid:3570 FATAL: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left on device
  <YYYY-MM-DD>T<time> UTC db: pid:3462 FATAL: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left on device
  <YYYY-MM-DD>T<time> UTC db: pid:3478 FATAL: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left on device
  <YYYY-MM-DD>T<time> UTC db: pid:4683 FATAL: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left on device
  <YYYY-MM-DD>T<time> UTC db: pid:5756 FATAL: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left on device
  </time></time></time></time></time>
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 5.1.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 5.0.x
VMware vCenter Server Appliance 5.5.x

Cause

This issue occurs if there is no space in the partition where the database resides.

Resolution

This is a known issue affecting vCenter Server 5.x.

Currently, there is no resolution.

To work around this issue, add a new VMDK disk, atleast 1.5 times the size of postgres database and copy the database to the new disk partition.

To add a new VMDK disk and copy the database:

1. Add a new VMDK, atleast 1.5 times the size of postgres database by using virtual machine edit settings option.
2. To initialize the disk:
   
   1. Log in to the appliance asg root You see output similar to:
      
      # df -h
      
      
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda3 9.8G 7.8G 2G 80% /
      udev 4.0G 104K 4.0G 1% /dev
      tmpfs 4.0G 0 4.0G 0% /dev/shm
      /dev/sda1 128M 21M 101M 18% /boot
      /dev/sdb1 20G 173M 19G 1% /storage/core
      /dev/sdb2 20G 2.3G 17G 12% /storage/log
      /dev/sdb3 60G 60G 0 100% /storage/db
      
      
   2. Browse to /dev and run ls | grep -i sd command:
      
      
      ls | grep -i sd
      
      
      sda
      sda1
      sda2
      sda3
      sdb
      sdb1
      sdb2
      sdb3
      sdc
      
      Note: sdc is the new device we attached in Step 1, if you do not see a new device, reboot the appliance.
      
      
      
   3. Run this command to format the new disk:
      
      
      fdisk /dev/sdc
      
      
      Notes:
      
      • To create a new partition, enter: n
      • To create the primary partition, enter: p
      • To create a partition with the specified settings, enter: w
      
      
   4. Run this command to create file system on the new device:
      
      mkfs -t ext3 /dev/sdc
      
      
   5. Create a mount point and mount the device by running these commands:
      
      
      mkdir /storage/sdc/
      mount /dev/sdc /storage/sdc/
      
      
   6. Check the output of df -h again:
      
      
      Filesystem Size Used Avail Use% Mounted on
      /dev/sda3 9.8G 7.8G 2G 80% /
      udev 4.0G 112K 4.0G 1% /dev
      tmpfs 4.0G 0 4.0G 0% /dev/shm
      /dev/sda1 128M 21M 101M 18% /boot
      /dev/sdb1 20G 173M 19G 1% /storage/core
      /dev/sdb2 20G 2.2G 17G 12% /storage/log
      /dev/sdb3 60G 60G 0 100% /storage/db
      /dev/sdc 99G 188M 94G 1% /storage/sdc
   
   
3. Stop the VMware Postgres service by running this command:
   
   service vmware-vpostgres stop
   
   
4. Copy the DB to the new disk partition:
   
   cp -ar /storage/db/vpostgres /storage/sdc/vpostgres
   
   
5. Start the database from the new location using this command:
   
   sudo -u postgres /opt/vmware/vpostgres/1.0/bin/pg_ctl -D /storage/sdc/vpostgres start
   
   
6. Find the postgres user password using this command:
   
   grep PGUSER_PASSWORD /etc/vmware-vpx/embedded_db.cfg
   
   
7. Run the vacuum command:
   
   

   sudo -u postgres /opt/vmware/vpostgres/1.0/bin/vacuumdb -a -e -v -f -U postgres > /tmp/vacuumdb.log