Extend root and alt_root volumes in SDDC Manager VM
Article ID: 400540
Updated On:
Products
Issue/Introduction
Low available free space in the root and/or alt_root partition in SDDC Manager.
Environment
This content is not related to a specific product version. It applies in case of low available free space in the root partition on the SDDC Manager VM.
Cause
The used space in the root volume of the SDDC Manager Appliance is reaching the total size which can cause system failure because of no free space. Increasing the root volume size must be in compliance with SDDC Manager upgrade process to avoid errors and failures during upgrade.
Resolution
If the root volumes size increase is necessary, specific requirements must be met in order to achieve this in compliance with SDDC Manager upgrade process. The newly added space must be on the same physical volume. This is the case for both root and alt_root volumes. alt_root volume is required to match the root volume size exactly in order to perform SDDC Manager upgrade. The steps will be done on the SDDC Manage VM requiring a few reboots.
Note : SDDC Manager VM must be backed up and VM snapshots taken right after the disks are resized.
- First the partition and logical volume used for root and alt_root in the system disk will be extended.
- Then the logical volume in the disk used for root swap will be extended as well to match in size.
Steps :
1. Navigate to vSphere UI and open sddc manager vm page then open 'Actions' > 'Edit Settings...'. Open hard disk menu in Virtual Hardware tab and expand the system and alt_root disks' total size to the desired number which must be larger than current size. The alt_root disk should be big enough to fit the size of the root logical volume (after the resize) and the lvm metadata. The first disk that should be expanded should be be Hard disk 1 and the other should be Hard disk 5. If there are any VM snapshots taken before that disk resize may be disabled.
2. Before the procedure lvm snapshots should be removed if they are no longer needed. If needed take backup and remove the snapshots.
Reboot SDDC Manager VM once the reconfigure VM task in vSphere has completed successfully and check that the system has detected the expanded disks. Check this with the 'lsblk' command. Use root user for all operations. (After the reboot the disks might be reorganized under different names).
Trigger a rescan of the SCSI device:
#echo 1 > /sys/class/block/sda/device/rescan
3. The system disk has 4 partitions and the last one is used for root or alt_root volumes. GPT must be updated to use the new unallocated space. Since it is the last partition on the disk, commands like 'parted' or 'fdisk' can be used to extend the partition with the new unallocated space at the end of the disk. It is advisable to validate the fstab UUID mounted in that partition before and after performing partition resize. After the resize, check the output of 'blkid' command and compare it to fstab. If the UUID for the logical volume in that partition is not the same for the corresponding mount point in /etc/fstab, update entry UUID in fstab to match.
Example:# parted /dev/<disk name> resizepart <partition number> 100%
'parted' command may prompt to fix GPT to use the new unallocated space instead if not done already. In that case, after the fix completes, it is required to run the command again to extend the partition.
4. Reboot SDDC Manager VM after the partition resize has completed successfully.
5. After the reboot the size of the volume group in that partition needs to be increased. Check the size in bytes for the partition using the 'lsblk -b' command and use it in the command below. The size must be smaller or equal to the size of the partition. Use 'pvresize' command to resize the volume group.
Example:# pvresize /dev/<partition name> --setphysicalvolumesize <new total size><size unit>
The 'setphysicalvolumesize' option can be omitted if you want to expand to the maximum available free space.
6. Now extend the logical volume and file system to the maximum available free space in this physical volume. Use the volume group and logical volume names that are in this partition. Use 'lvextend' command to resize the logical volume.
Example:# lvextend -r -l +100%FREE /dev/<volume group name>/<logical volume name>
'-r' will also resize the ext4 filesystem as well.
'-l +100%FREE' will resize the logical volume to the maximum available free space in this volume group.
7. Next we will resize the logical volume mounted at second disk that we expanded. This disk does not have partitions, only lvm group and volume. The new size of the volume group must be the same used in step 5. Check the size in bytes for the partition using the 'lsblk -b' command and use it in the command below. The size of the disk should be big enough to fit the new size. Use 'pvresize' command to resize the volume group.
Example:# pvresize /dev/<disk name> --setphysicalvolumesize <new total size><size unit>
8. Finally its the file system that needs to be extended to the maximum available free space in this physical volume. Use the volume group and logical volume names that are in this disk. Use 'lvextend' command to resize the logical volume.
Example:# lvextend -r -l +100%FREE /dev/<volume group name>/<logical volume name>
'-r' will also resize the ext4 filesystem as well.
'-l +100%FREE' will resize the logical volume to the maximum available free space in this volume group.
9. Check with 'lsblk -b' command if the sizes of root and alt_root volumes match now.
Feedback
