Upgrading from VMware Aria Suite Lifecycle 8.12.x and above may fail to start with "Operation not allowed in the current state of workflow" exception

Products

• VMware Aria Suite

Issue/Introduction

This is applicable for upgrades from 8.12 and above versions of VMware Aria Suite Lifecycle where the Common Appliance Platform (CAP) is leveraged to handle and manage upgrades.

Symptoms:

• You are trying to upgrade VMware Aria Suite Lifecycle 8.12.x and above but it immediately fails before initiating the upgrade to the new version.
• You find similar log entries in the /var/log/vmware/capengine/core-engine.log

2024/03/01 15:01:50.652977 workflow_handler.go:168: Starting workflow cap-non-lvm-update
2024/03/01 15:01:50.653043 workflow_handler.go:154: Error in executing request. Error: Operation not allowed in the current state of workflow
2024/03/01 15:01:50.653049 engine.go:127: Failed to execute workflow request. Error: Operation not allowed in the current state of workflow

Environment

VMware Aria Suite Lifecycle 8.14.x
VMware Aria Suite Lifecycle 8.12.x

Resolution

VMware is aware of this issue. See the Workaround section below for additional information.

Workaround:

Prerequisites

• You have valid backups or snapshots of the Aria Suite Lifecycle appliance.
• You have root username and password for the Aria Suite Lifecycle appliance
• You have access to SSH or PuTTy.

Procedure

1. SSH into the Aria Suite Lifecycle appliance.
2. Browse to the /storage/lifecycle/capengine directory and validate a file workflow.db exists.

   /storage/lifecycle/capengine ls -ltrh
   -rw-r--r-- 1 root root 36K Mar  2 20:47 workflow.db

3. Check the status of cap-workflow-engine service by running the below command:

   systemctl status cap-workflow-engine.service

4. Stop the cap-workflow-engine.service:

   systemctl stop cap-workflow-engine.service

5. Rename cap engine service workflow database by running below command:

   mv /storage/lifecycle/capengine/workflow.db /storage/lifecycle/capengine/workflow.db.bak

6. Start the cap-workflow-engine.service:

   systemctl start cap-workflow-engine.service

Note: This will recreate a new workflow.db file under the same location.

7. Check the status of cap engine and make sure it is in running state:

systemctl status cap-workflow-engine.service

Example output:

systemctl status cap-workflow-engine.service
* cap-workflow-engine.service - Cap workflow engine
   Loaded: loaded (/lib/systemd/system/cap-workflow-engine.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2024-03-03 02:22:18 UTC; 41min ago
 Main PID: 2981 (bash)
    Tasks: 7 (limit: 5883)
   Memory: 181.5M
   CGroup: /system.slice/cap-workflow-engine.service
           |-2981 /bin/bash -c . /etc/profile; /usr/lib/vmware-capengine/capengine
           `-3039 /usr/lib/vmware-capengine/capengine

Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82> Handlers ............ 10  Threads ............. 2 <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82> Prefork ....... Disabled  PID .............. 3039 <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><94><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><8C><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94><80><E2><94>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82>                    Fiber v2.1.0                   <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82>            http:///tmp/capengine.sock:            <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82>                                                   <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82> Handlers ............ 10  Threads ............. 2 <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><82> Prefork ....... Disabled  PID .............. 3039 <E2><94><82>
Mar 03 02:22:21 vasl_appliance_fqdn bash[2981]:  <E2><94><94><E2><94><80><E2><94><80><E2><94><80>

8. Retry the upgrade.