Reclaiming datastore space from thin provisioned vCenter Server Appliance

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter Server Appliance (VCSA) is set to thin provisioned for storage on VMFS6.
  
  
• With VMFS6, unmap is automatically enabled, which for thin provisioned VM's is suppose to recover empty blocks at the end of the disk.
  
  
• After investigating the space usage in the vCenter UI, you find that the disk usage in the guest OS is smaller than the disk usage reported in the vCenter UI.

Cause

The space first needs to be zeroed out before it can be reclaimed. More information can be read about this in the following article:

Reclaiming disk space from thin provisioned VMDK files when ESXI Host still sees blocks even after removed from GuestOS

Resolution

The following steps need to be taken on the base disks of the VCSA. Due to this, taking a snapshot will invalidate the following process so a snapshot of the VCSA CANNOT be taken as a rollback option. The following options are still available:

• A VAMI backup
• 3rd party backup (VM level backup)
• Clone the VCSA 

Make sure that one or multiple of these are available before beginning the below steps.

1. Stop services on the VCSA:
   
   service-control --stop
   
   
2.  Run the dd command against the majority of the drives:
   
   dd if=/dev/zero of=/storage/lvm_snapshot/zeroes
dd if=/dev/zero of=/storage/lifecycle/zeroes
dd if=/dev/zero of=/storage/autodeploy/zeroes
dd if=/dev/zero of=/storage/dblog/zeroes
dd if=/dev/zero of=/storage/db/zeroes
dd if=/dev/zero of=/storage/seat/zeroes
dd if=/dev/zero of=/storage/updatemgr/zeroes
dd if=/dev/zero of=/storage/vtsdblog/zeroes
dd if=/dev/zero of=/storage/imagebuilder/zeroes
dd if=/dev/zero of=/storage/core/zeroes
dd if=/dev/zero of=/storage/log/zeroes
dd if=/dev/zero of=/storage/vtsdb/zeroes
dd if=/dev/zero of=/tmp/zeroes
      
    Drives excluded:
    / - The root partition filling up it normally stops the appliance from running
    /storage/archive - This drive normally fills up to 99% and stays there by design. This drive normally has no free space to recover.
    /storage/netdump -  This drive is 1 GB in size, which is a nominal space savings.
   
   
3. Remove the zeroes files that were created on the last step:
   
   (Note: rm is a data destructive command. The absolute path has been specified with the correct file name. Double check your paths are absolute and your filenames are correct)
   
   rm -f /storage/lvm_snapshot/zeroes
rm -f /storage/lifecycle/zeroes
rm -f /storage/autodeploy/zeroes
rm -f /storage/dblog/zeroes
rm -f /storage/db/zeroes
rm -f /storage/seat/zeroes
rm -f /storage/updatemgr/zeroes
rm -f /storage/vtsdblog/zeroes
rm -f /storage/imagebuilder/zeroes
rm -f /storage/core/zeroes
rm -f /storage/log/zeroes
rm -f /storage/vtsdb/zeroes
rm -f /tmp/zeroes
   
   
4. Start services on the VCSA:
   
   service-control --start
   
   
5. Migrate the VCSA to another datastore, changing the storage type from Thin Provisioned to Thick Provisioned. (Note: Eager or Lazy Zero should not matter)
   
   
6. After the storage migration, power off the VCSA.
   
   
7. Once the VCSA is powered off completely, migrate the VCSA back to the source datastore changing from Thick Provisioned to Thin Provisioned.

Additional Information

After following the above steps, the disk usage in the UI will still be slightly above what the disk usage in the guest OS. This is due to the drives that were excluded.