VCF Operations Cloud Proxy upgrade from 9.0.2 to 9.1.x stuck at 56%

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

•  When attempting to upgrade VCF Operations (formerly Aria Operations) from version 9.0.2 to 9.1.x, the Analytics node upgrades successfully, but the Cloud Proxy upgrade stalls at 56%.

• The automated upgrade fails to progress, and the payload download phase never commences.

• Checking the cprc-cli --status on the Cloud Proxy shows the Download status as [NONE] and Downloaded PAK size as [] MB.

  root@<Cloudproxy_FQDN> [ ~ ]# cprc-cli --status
  -----------------------------------------------------------------------
  Product Upgrade
  -----------------------------------------------------------------------
      Automated upgrade enabled: [False]
      Status: [RUNNING]
      Product PAK: [Operations-Cloud-Proxy-Upgrade-910010025434833]
      Started: [<Date&Time>]
      Download status: [NONE]
      Download start: [<Date&Time>]
      Downloaded PAK size: [] MB
      Download end: []
  -----------------------------------------------------------------------
  Product Latest Upgrade
  -----------------------------------------------------------------------
      Product PAK: [Operations-Cloud-Proxy-Upgrade-910010025434833]
      Status: [RUNNING]
      Type: [AUTOMATED]
      Started: [<Date&Time>]
      Completed: []
      Extract status: [NONE]
      Download status: [NONE]
      Upgrade system reboot at: []

• The cprc.upgrade.status file on the affected Cloud Proxy indicates that the certificate_renew_status is stuck in a RUNNING state with no certificate_renew_end_time recorded.

 

Environment

VCF Operations 9.1.x

Cause

This issue occurs due to a stale certificate_renew_status flag left over from a previously interrupted operation. If a certificate renewal was initiated (e.g., months prior) but the process was abruptly interrupted (such as a node reboot or killed process) before completion, the status remains flagged as RUNNING.

Because the system currently lacks a timeout or crash-recovery guard mechanism for this specific renewal flow, all subsequent automated upgrade attempts read this stale status. To prevent conflicts, the upgrade workflow halts and blocks the download phase from initiating.

Resolution

To resolve this issue, run the built-in CPRC Recovery Script on the affected Cloud Proxy appliance. This is the recommended method to clear the blocked status.

1. Open an SSH session to the affected Cloud Proxy appliance and log in as root.

2. Execute the following command to run the recovery script:

   python3 /usr/lib/vmware-vrops-cprc/bin/python/cprcrecovery/cprcRecovery.py

3. The script will automatically perform the following operations:

   • Reset the stale upgrade and certificate statuses.

   • Set auto_upgrade_check_enabled back to true in the cprc.auto.upgrade.configuration file.

   • Trigger the next upgrade attempt automatically once the configuration is fully restored.