Upgrade fails during "Perform Component Upgrade" step with internal error in VCF Automation

Products

• VCF Automation

Issue/Introduction

1. When upgrading VCF Automation from version 9.0.x to 9.1, the upgrade process may fail during the Perform Component Upgrade phase with the following error in the UI:

   Internal error occurred: Please retry the operation or contact support if the issue persists

2. The UI offers no option to retry, and the VCF Automation UI becomes completely inaccessible following the failed attempt.
3. This issue will display one or more of the following symptoms:
   • The VCF Automation UI becomes inaccessible after the upgrade attempt fails.

• • Tenant manager pods - tenant-manager-# fails to initialize.

    prelude tenant-manager-0 0/1 Init:0/2 0 

  • The tenant-manager-configure pod shows a status of Error.

    prelude tenant-manager-configure-#### 0/1 Error 0

     

  • Logs from the tenant-manager-configure pod show the following:

    YYYY-MM-DD HH:MM:SS,### | INFO | main | c.v.v.u.l.UpgradeAgent | Examining database at URL: jdbc:postgresql://vcfapostgres.prelude.svc.cluster.local:5432/tenantmanager?socketTimeout=90&ssl=verify-full&sslrootcert=/vmsp-platform-trust/bundle.pem | %PARSER_ERROR[x]
    YYYY-MM-DD HH:MM:SS,### | ERROR | main | c.v.v.u.l.UpgradeAgent | A previous attempt to upgrade the database was incomplete. Please restore your database from a backed up copy and try again. | %PARSER_ERROR[x]

Environment

VCF Automation 9.0.x

Cause

1. A defect in the Tenant Manager's periodic scheduler prevents background purge and cleanup jobs from resuming after a cell restart or failover. Because stale execution times are no longer re-evaluated, routine database cleanup tasks stop running entirely.
2. As a result, the user_session and session_role tables grow excessively large over time. During an upgrade to VCF Automation 9.1, this table inflation exceeds internal thresholds and causes the Tenant Manager database schema migration to time out, fail, or prevent the pod from initializing.

Resolution

A fix for the scheduler is included in the future release. For assistance with rollback and database truncation, contact Broadcom Support.

Workaround:

If you have not yet attempted the upgrade or need to recover and retry, perform the following steps prior to triggering the upgrade to VCF Automation 9.1:

1. Connect to the Tenant Manager database instance.
2. Observe the number of records in the following tables:

   select count(*) from user_session;
   select count(*) from session_role;

3. Truncate the two tables to clean them up:

   TRUNCATE user_session,session_role;

4. Observe the number of records again in the tables to confirm they are cleaned up.
5. Proceed with the upgrade from VCF Automation 9.0.x to 9.1.

Additional Information

Attempting an upgrade while tables are inflated will result in downtime and an inaccessible VCFA management console, requiring a database restore to recover. Always ensure valid database backups exist prior to starting system upgrades.

Reference TechDocs: Component Backup and Restore of VMware Cloud Foundation