VMware Cloud Foundation /data partition where postgres saves data gets filled up
search cancel

VMware Cloud Foundation /data partition where postgres saves data gets filled up

book

Article ID: 311989

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:
  • SDDC Manager UI is not working.
  • On restarting SDDC Manager, SDDC Manager services don't come up because of postgres related problems.
  • postgres startup fails.
  • The device mounted on filesystem mount point - /dev/mapper/data_vg-data - is full




Environment

VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 5.0, 5.1

Resolution

Currently there is no resolution.


Workaround:

To workaround this issue, please follow the below steps:


For VCF versions below 4.5

1. Extend the /data partition by following these steps:

  • Power off the SDDC Manager VM and wait for a few moments until it's properly turned off.

  • Add a new disk to the VM as per below.

  • Turn on the SDDC Manager VM and wait for it to properly come back up.


     

    2. Go to the SDDC Manager VM and run the following commands

  •  echo "- - -" > /sys/class/scsi_host/host2/scan

  •  lsblk

In your lsblk output you should see the new disk added.

In this example, I added a new disk with 16gb and it's denoted as sdg.

Also please note which disk your /data partition is located under. In this case mine is under sdc. As well as the path to the /data directory. In the screenshot mine is vg_data-lv_data1

3. Add the new disk

  • pvcreate /dev/sdg      <---- Use the newly added disk

  • Find out which Volume Group that your disk belongs to by running
    pvs

4. Extend the Volume Group with the New Disk

  • vgextend vg_data /dev/sdg  <---- Use the newly added disk and VG

  • lvdisplay vg_data ← In this case I'm looking for lv_data1 and I want to copy the path

Extend the space using the path

5.  lvextend -L+16G -r /dev/vg_data/lv_data1 . ← Might have to adjust the size to be less than 16G. Can specify what you want

6. You can verify with df -h that the /data directory now has grown in size.

 

For VCF versions 4.5 and above

1. Extend the /data partition by following these steps:

  • Power off the SDDC Manager VM and wait for a few moments until it's properly turned off.

  • Add a new disk to the VM as per below.

  • Turn on the SDDC Manager VM and wait for it to properly come back up.


     

    2. Go to the SDDC Manager VM and run the following commands

  •  echo "- - -" > /sys/class/scsi_host/host2/scan

  •  lsblk

In your lsblk output you should see the new disk added.

In this example, I added a new disk with 16gb and it's denoted as sdg.

Also please note which disk your /data partition is located under. In this case mine is under sdb. As well as the path to the /data directory. In the screenshot mine is data_vg-data

3. Add the new disk

  • pvcreate /dev/sdg      <---- Use the newly added disk

  • Find out which Volume Group that your disk belongs to by running
    pvs

4. Extend the Volume Group with the New Disk

  • vgextend data_vg /dev/sdg  <---- Use the newly added disk and VG

  • lvdisplay data_vg ← In this case I'm looking for data and I want to copy the path

Extend the space using the path

5.  lvextend -L+16380M -r /dev/data_vg/data . ← Might have to adjust the size to be less than 16G. Can specify what you want

6. Sometimes a resize is needed:  resize2fs /dev/mapper/data_vg-data

6. You can verify with df -h that the /data directory now has grown in size.