VMware Cloud Foundation /data partition where postgres saves data gets filled up

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• SDDC Manager UI is not working.
• On restarting SDDC Manager, SDDC Manager services don't come up because of postgres related problems.
• postgres startup fails.
• The device mounted on filesystem mount point - /dev/mapper/data_vg-data - is full

Environment

VMware Cloud Foundation 4.0.x

VMware Cloud Foundation 5.0, 5.1

Resolution

Currently there is no resolution.

Workaround:

To workaround this issue, please follow the below steps:

Note: Take Snapshot of the SDDC VM prior to executing the disk expansion steps. 


For VCF versions below 4.5

1. Extend the /data partition by following these steps:

• Power off the SDDC Manager VM and wait for a few moments until it's properly turned off.

• Add a new disk to the VM as per below.

• Turn on the SDDC Manager VM and wait for it to properly come back up.
  
  
   

  2. Go to the SDDC Manager VM and run the following commands

•  echo "- - -" > /sys/class/scsi_host/host2/scan

•  lsblk

In your lsblk output you should see the new disk added.

In this example, I added a new disk with 16gb and it's denoted as sdg.

Also please note which disk your /data partition is located under. In this case mine is under sdc. As well as the path to the /data directory. In the screenshot mine is vg_data-lv_data1

3. Add the new disk

• pvcreate /dev/sdg      <---- Use the newly added disk

• Find out which Volume Group that your disk belongs to by running
  pvs

4. Extend the Volume Group with the New Disk

• vgextend vg_data /dev/sdg  <---- Use the newly added disk and VG

• lvdisplay vg_data ← In this case I'm looking for lv_data1 and I want to copy the path

Extend the space using the path

5.  lvextend -L+16G -r /dev/vg_data/lv_data1 . ← Might have to adjust the size to be less than 16G. Can specify what you want

6. You can verify with df -h that the /data directory now has grown in size.

 

For VCF versions 4.5 and above

1. Extend the /data partition by following these steps:

• Power off the SDDC Manager VM and wait for a few moments until it's properly turned off.

• Add a new disk to the VM as per below.

• Turn on the SDDC Manager VM and wait for it to properly come back up.
  
  
   

  2. Go to the SDDC Manager VM and run the following commands

•  echo "- - -" > /sys/class/scsi_host/host2/scan

•  lsblk

In your lsblk output you should see the new disk added.

In this example, I added a new disk with 16gb and it's denoted as sdg.

Also please note which disk your /data partition is located under. In this case mine is under sdb. As well as the path to the /data directory. In the screenshot mine is data_vg-data

3. Add the new disk

• pvcreate /dev/sdg      <---- Use the newly added disk

• Find out which Volume Group that your disk belongs to by running
  pvs

4. Extend the Volume Group with the New Disk

• vgextend data_vg /dev/sdg  <---- Use the newly added disk and VG

• lvdisplay data_vg ← In this case I'm looking for data and I want to copy the path

Extend the space using the path

5.  lvextend -L+16380M -r /dev/data_vg/data . ← Might have to adjust the size to be less than 16G. Can specify what you want

6. Sometimes a resize is needed:  resize2fs /dev/mapper/data_vg-data

6. You can verify with df -h that the /data directory now has grown in size.

Additional Information

Note: If the disk extension has already been done and when extending the disk second time , we have to use the ' New' disk name ,

It will be sdh in the vgextend command, which can be seen in the blkid command output. 

vgextend data_vg /dev/sdh