Failed to update VCF Services and Photon rpms in SDDC Manager

Products

• VMware SDDC Manager

Issue/Introduction

Upgrading SDDC manager to 5.2 fails at the below step - 

You find the below error traces in -

• /var/log/vmware/vcf/lcm/thirdparty/upgrades/<upgrade_bundle_id>/vcf-platform/upgrade/vcf_platform_upgrade.log:

2024-11-07 12:25:49,298: ERROR: Starting task reclaim-vfree
2024-11-07 12:25:49,298: INFO: Updated /var/log/vmware/vcf/lcm/thirdparty/upgrades/5791cec9-dc88-40a3-8129-b8767ac8493b/vcf-platform/upgrade/vcf_platform_upgrade.status status file with data OrderedDict
([('upgradeId', '5791cec9-dc88-40a3-8129-b8767ac8493b'), ('resourceId', '4d3f2864-8f03-4b0d-9c63-c59793eb3687'), ('upgradeStatusCode', 'COMPLETED_WITH_FAILURE'), ('progress', 20), ('error', OrderedDict([('errorCode', 200), ('errorDescription', 'Starting task reclaim-vfree')])), ('startTime', 1730981868), ('endTime', 1730982349)])
2024-11-07 12:25:49,298: ERROR:
Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b5ef5cb1-7dee-4b11-8abf-1dc42022662f/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 472, in <module>
    upgrade_workflow()
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b5ef5cb1-7dee-4b11-8abf-1dc42022662f/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 119, in upgrade_workflow
    wrapper.update_status(return_code=rc, status='COMPLETED_WITH_FAILURE',
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b5ef5cb1-7dee-4b11-8abf-1dc42022662f/thirdparty/vcf-platform-upgrade/bin/../../wrapper.py", line 187, in update_status
    raise Exception
Exception
2024-11-07 12:25:49,300: INFO: URL: http://localhost/lcm/about

2024-11-07 12:25:49,313: ERROR: Failed to update VCF Services and Photon rpms in SDDC Manager
2024-11-07 12:25:49,365: INFO:
2024-11-07 12:25:49,365: INFO: RC: 1, OUT:
2024-11-07 12:25:49,365: INFO: ERR: Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b5ef5cb1-7dee-4b11-8abf-1dc42022662f/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 472, in <module>
    upgrade_workflow()
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b5ef5cb1-7dee-4b11-8abf-1dc42022662f/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 119, in upgrade_workflow
    wrapper.update_status(return_code=rc, status='COMPLETED_WITH_FAILURE',
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b5ef5cb1-7dee-4b11-8abf-1dc42022662f/thirdparty/vcf-platform-upgrade/bin/../../wrapper.py", line 187, in update_status
    raise Exception
Exception

• 
  var/log/vmware/capengine/cap-update/install-<Time_&_Date>:

installing package rsync-3.2.7-1.ph4.Error(1525) : rpm transaction failed
x86_64 needs 49MB more space on the / filesystem

installing package glibc-devel-2.32-16.ph4.x86_64 needs 71MB more space on the / filesystem
installing package c-ares-1.19.1-2.ph4.x86_64 needs 58MB more space on the / filesystem
installing package linux-5.10.214-1.ph4.x86_64 needs 84MB more space on the / filesystem
installing package selinux-policy-36.5-4.ph4.noarch needs 90MB more space on the / filesystem
Error in upgrading all packages to latest versions (tdnf error code: 245).
Original list of RPMs and RPM DB are stored in /tmp/photon-upgrade-J1P,  please provide contents of that folder along with system journal   logs for analysis; these logs can be captured using command-
 # /usr/bin/journalctl -xa > /tmp/photon-upgrade-J1P/journal.log

Environment

VMware Cloud Foundation 5.2.x

Cause

SDDC manager has a partition that at 100%, (or almost full).  The error - x86_64 needs 49MB more space on the / filesystem indicates root partition is full.

Resolution

1. SSH with vcf user to the SDDC manager, elevate to root user.

2. Run the below command  to confirm which partition is full.

df -h

3. Navigate around each directory to identify the folders/logs taking more space using the below commands - 

du -h /--max-depth 1
du -sh * | sort -nr

4. After clearing the space, reboot the SDDC manager and re-try the update.