SDDC Manager upgrade fails at - Update VCF Service and Platform rmps: Starting task reclaim-vfree

Products

• VMware SDDC Manager

Issue/Introduction

• Upgrading SDDC manager to 5.2 fails at the below step :

• In the SDDC manager UI, the upgrade failure displays the error: 'Starting task reclaim-vfree'.

• /var/log/vmware/vcf/lcm/thirdparty/upgrades/<upgrade_bundle_id>/vcf-platform/upgrade/vcf_platform_upgrade.log:

yyyy-mm-dd hh:mm:ss: ERROR: Starting task reclaim-vfree
yyyy-mm-dd hh:mm:ss: INFO: Updated /var/log/vmware/vcf/lcm/thirdparty/upgrades/####/vcf-platform/upgrade/vcf_platform_upgrade.status status file with data OrderedDict
([('upgradeId', '####'), ('resourceId', '####'), ('upgradeStatusCode', 'COMPLETED_WITH_FAILURE'), ('progress', 20), ('error', OrderedDict([('errorCode', 200), ('errorDescription', 'Starting task reclaim-vfree')])), ('startTime', xxx), ('endTime', xxx)])
yyyy-mm-dd hh:mm:ss: ERROR:
Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/####/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 472, in <module>
    upgrade_workflow()
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/####/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 119, in upgrade_workflow
    wrapper.update_status(return_code=rc, status='COMPLETED_WITH_FAILURE',
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/####/thirdparty/vcf-platform-upgrade/bin/../../wrapper.py", line 187, in update_status
    raise Exception
Exception
yyyy-mm-dd hh:mm:ss: ERROR: Failed to update VCF Services and Photon rpms in SDDC Manager

• /var/log/vmware/capengine/cap-update/install-time_&_date:

installing package rsync-3.2.7-1.ph4.Error(1525) : rpm transaction failed
x86_64 needs 49MB more space on the / filesystem

installing package glibc-devel-2.32-16.ph4.x86_64 needs 71MB more space on the / filesystem
installing package c-ares-1.19.1-2.ph4.x86_64 needs 58MB more space on the / filesystem
installing package linux-5.10.214-1.ph4.x86_64 needs 84MB more space on the / filesystem
installing package selinux-policy-36.5-4.ph4.noarch needs 90MB more space on the / filesystem
Error in upgrading all packages to latest versions (tdnf error code: 245).

Environment

VMware Cloud Foundation 5.2.x

Cause

• SDDC manager has a partition at 100%, (or almost full).
  
  
• The error in this case - x86_64 needs 49MB more space on the / filesystem, indicates root partition is full.

Resolution

1. SSH with vcf user to the SDDC manager, elevate to root user.

2. Run the below command to identify which partition is full.

df -h

3. Locate Space-Consuming Files/Folders - navigate around each directory to methodically identify the directories and log files consuming the most space:

• Run this command from the root of the full partition to see the total size of each immediate subdirectory

du -h --max-depth 1 /<Mount_Point> # Example: du -h --max-depth 1 /

• Sort Files/Folders by Size

du -sh * | sort -nr

4. After successfully clearing the required disk space, reboot the SDDC manager and re-try the update.

• Verify that the partition is no longer full by running df -h again.
• Reboot the SDDC Manager virtual appliance to ensure all services restart cleanly.
• Re-attempt the Upgrade.