SDDC Manager Upgrade to 5.2.4.0 Fails at VCF Platform Services Due to Timeout
search cancel

SDDC Manager Upgrade to 5.2.4.0 Fails at VCF Platform Services Due to Timeout

book

Article ID: 452426

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • VMware Cloud Foundation (VCF) SDDC Manager upgrade to target version 5.2.4.0 fails at the "VCF Platform Services" stage due to a timeout. The upgrade process hangs and eventually exceeds the 60-minute execution window.
  • The /var/log/vmware/vcf/lcm/thirdparty/upgrade/upgrade_id/vcf_platform_upgrade.log indicates that the process is stuck executing the following ansible-playbook command:

YYY-MM-DDTHH:MM:SS.Z: INFO: RC: 0
YYY-MM-DDTHH:MM:SS.Z: INFO: Execute cmd: ansible-playbook --connection=local -i <REDACTED_IP>, /usr/share/ansible/stig-hardening/playbook.yml -v --tags PHTN-40-000004,PHTN-40-000012,PHTN-40-000031,PHTN-40-000035,PHTN-40-000036,PHTN-40-000037,PHTN-40-000038,PHTN-40-000042,PHTN-40-000043,PHTN-40-000074,PHTN-40-000086,PHTN-40-000093,PHTN-40-000108,PHTN-40-000110,PHTN-40-000112,PHTN-40-000173,PHTN-40-000175,PHTN-40-000184,PHTN-40-000192,PHTN-40-000193,PHTN-40-000194,PHTN-40-000196,PHTN-40-000219,PHTN-40-000235,PHTN-40-000237,PHTN-40-000245,PHTN-40-000246

Environment

SDDC Manager 5.2.x

VCF 9.x 

Cause

The failure is caused by a resource-contention race condition between the internal aide --check STIG hardening scan and the vcf-platform-upgrade OS migration. Both processes perform intensive I/O operations concurrently on the same filesystem directories (/boot, /opt, /usr, /etc, and /var/log), generating massive disk I/O contention.

Resolution

This is a known issue and Engineering is actively looking to resolve this issue. 

Workaround: 

Make sure to have a snapshot of the sddc manager before performing the workaround 

  1. Identify the hung ansible-playbook processes associated with the upgrade on the SDDC Manager appliance, using ps aux | grep ansible

  2. Manually terminate the identified processes to release the file locks by running kill -9 process-id 

  3. Allow the vcf-platform-upgrade orchestrator to resume and complete the operation successfully.

Powered by Wolken Software