VCF Operations upgrade stalls at "Step 7 of 14 - Applying operating system update" with tdnf Error 1037 - VCF Operations 9.1.1

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When attempting to upgrade VCF Operations, the upgrade process stalls indefinitely at "Step 7 of 14 - Applying operating system update".

Symptoms include:

• The Admin UI gets stuck in a "ghost task" state, where the upgrade progress does not proceed past Step 7.

• Cloud Proxies or other cluster nodes may silently drop from the active upgrade sequence, and the remediation option disappears from the Admin UI.

• In the primary node console or the OS upgrade logs (/tmp/photon-upgrade-*/journal.log), the following errors are observed indicating a fatal package manager crash:
  
  (re)generate initramfs for all kernels,
ERROR: duplicate repo id: photon-extras
Error(1037) : Duplicate repo id
YY-MM-dateT13:00:45+0000 Could not erase all unsupported packages (tdnf error code: 13).
Cannot continue. Aborting.

Environment

 

• VMware Cloud Foundation (VCF) Operations 9.1.1

• VMware Cloud Foundation (VCF) Operations 9.0.2 / 9.1.0 (Source Versions)

 

Cause

This issue is caused by two compounding factors during the upgrade sequence:

1. Duplicate Repositories (Step 7 Failure): A duplicate [photon-extras] repository definition exists in the /etc/yum.repos.d/ directory (e.g., it is defined in both photon.repo and a standalone photon-extras.repo). This causes the Photon OS package manager (tdnf) to crash with a fatal Error 1037 during the OS update phase. This aborts the update script before it can generate the new kernel's initramfs, leaving the CaSA orchestrator in a hung state without a formal failure code.

2. Expired Root Passwords (Proxy/Node Failure): The local root account passwords have expired according to the OS chage policy. This blocks the main cluster from authenticating via SSH to push the upgrade payload to the remaining nodes or proxies, causing them to drop from the active upgrade sequence.

Resolution

To resolve this issue, you must remove the duplicate repository definitions, clear the hung orchestrator state, reset the expired passwords, and re-initiate the upgrade.

Part 1: Resolve the Duplicate Repository

1. Open an SSH session (or virtual machine console) to all affected cluster nodes.

2. Search for the duplicate repository by running the following command: grep -r "photon-extras" /etc/yum.repos.d/

3. If duplicates are found, move the standalone repository file out of the directory: mv /etc/yum.repos.d/photon-extras.repo /tmp/

4. Verify the resolution by confirming this command runs cleanly without throwing Error 1037: tdnf repolist --all

Part 2: Clear the CaSA Orchestrator State

1. Clear the stuck CaSA orchestrator history by moving the JSON files to /tmp/: mv /storage/log/vcops/log/casa/casa_pak_history_*.json /tmp/

2. Restart the CaSA service to unlock the Admin UI: systemctl restart vmware-casa 
   
   Note : The upgrade orchestrator will automatically reboot the nodes to apply the new OS kernel. This automated reboot is expected behavior.

Part 3: Re-initiate the Upgrade

1. Log into the VCF Operations Admin UI.

2. Check the upgrade process should now successfully progress past Step 7.

Additional Information

Do NOT manually reboot nodes while Step 7 is stalled. Because the update script aborts before generating the initramfs for the new kernel, manually rebooting the virtual machine from vCenter in this state will result in a kernel panic and an unbootable appliance. Allow the automated orchestrator to handle the reboot once the upgrade resumes.