Aria Automation 8.18.x Upgrade Fails with "Timeout expired Deployment failed" due to vami-sfcb Service Startup Issue (Outdated Python Scripts)

Products

• VMware Aria Suite

Issue/Introduction

When attempting to upgrade Broadcom Aria Automation from version 8.18.0 to 8.18.1, the upgrade process fails during the initial startup phase with a "Timeout expired Deployment failed" error. This failure is specifically linked to the vami-sfcb service failing to start.

Symptoms

You will observe the following symptoms during the upgrade attempt:

1. The vracli status first-boot command (executed internally during upgrade) reports a failure for the vami-sfcb service check:
   ++ source /opt/scripts/retry_utils.sh
+ vracli status first-boot -w 300 failed check /usr/bin/systemctl is-active vami-sfcb: exit code: [3], stdout: [failed ], stderr: []
+ die 'Timeout expired'
+ local 'msg=Timeout expired'
+ local exit_code=
+ '[' 1 -lt 2 ']'
+ exit_code=1
+ set +x
^[[H^[[2J^[[3JTimeout expired
Deployment failed. Collecting log bundle ...


2. Manually checking the vami-sfcb service status on the Aria Automation appliance shows it in a failed state
   systemctl status vami-sfcb.service

Environment

Aria Automation 8.18.x

Cause

The root cause of this issue is the presence of outdated VAMI Python library files in the /opt/vmware/share/vami directory on the Aria Automation appliance. Specifically, files such as vami_login_textgen, vami_invoke_method, vami_sfcb_test, and vami_sfcb_initialize still contain shebangs pointing to #!/usr/bin/env python2.

Aria Automation versions 8.16.1 and later require these scripts to use #!/usr/bin/env python, which resolves to Python 3. The vami-sfcb service relies on these scripts for its proper operation. When the upgrade process attempts to start the vami-sfcb service, these outdated scripts fail to execute correctly due to Python 2 deprecation or absence in the updated environment. This failure causes the vami-sfcb service to time out, blocking the upgrade.

 

Resolution

To resolve this issue and allow the Aria Automation upgrade to complete, you need to manually update the VAMI RPM packages to ensure the correct Python 3 compatible scripts are in place.

Steps:

1. Obtain Correct VAMI RPMs: Download the specific VAMI RPM packages for Aria Automation 8.18.1 Patch 2 (Build 24862833) or the latest equivalent from the Broadcom Support Portal.
   
   The required RPMs are:
   vmware-studio-vami-cimom_3.0.0.7-24345548.x86_64.rpm
vmware-studio-vami-login_3.0.0.7-24345548.x86_64.rpm
vmware-studio-vami-tools_3.0.0.7-24345548.x86_64.rpm
   
   Note: Ensure you download the packages corresponding to your exact Aria Automation 8.18.x version and patch level, or as advised by Broadcom Support.
   
   
2. Transfer RPMs to Appliance: Transfer the downloaded RPM files to the /tmp directory (or another temporary location) on your Aria Automation appliance. You can use tools like scp.
3. SSH to Appliance: Access the Aria Automation appliance via SSH as root.
4. Install the RPMs: Execute the following commands to install the updated VAMI RPMs. This will force an overwrite of the existing outdated files.
   
   rpm -Uvh --nodigest --nofiledigest --force /tmp/vmware-studio-vami-cimom_3.0.0.7-24345548.x86_64.rpm
rpm -Uvh --nodigest --nofiledigest --force /tmp/vmware-studio-vami-login_3.0.0.7-24345548.x86_64.rpm
rpm -Uvh --nodigest --nofiledigest --force /tmp/vmware-studio-vami-tools_3.0.0.7-24345548.x86_64.rpm
   
   Replace /tmp/ with the actual path if you placed them elsewhere.
   
   
5. Verify vami-sfcb service status: After installing the RPMs, the vami-sfcb service should now start correctly. You can manually check its status:
   
   systemctl status vami-sfcb.service
   
   The output should now show Active: active (running). If it's not running, try starting it with systemctl start vami-sfcb.service.
   
   
6. Resume Upgrade: Once the vami-sfcb service is confirmed to be running, you can retry or continue your Aria Automation upgrade process. The update should now proceed without the vami-sfcb timeout error.