VMware VeloCloud SD-WAN Orchestrator upgrade steps from any version starting from 4.x to 5.x.

Products

VMware VeloCloud SD-WAN

Issue/Introduction

VCO upgrade steps to upgrade VCO from from any version starting from 4.x to 5.x.

This procedure is not valid for upgrade from 3.x version to higher versions as well as upgrade to 6.x version.

Environment

VMware SDWAN Orchestrator

Cause

NA

Resolution

Pre-requisites before starting VCO upgrade :-

Prepare for VCO Upgrade (Follow the VMware SASE Orchestrator Deployment and Monitoring Guide)
Send upgrade announcement from Orchestrator Upgrade screen on VCO

All commands presented here are intended to be executed by "root"
While the process can be executed over SSH (and might be the only option in certain environments), it is recommended to be done from the VM console to avoid potential issues that might happen if the SSH connection is interrupted for any reason.
You may obtain software images from "https://support.broadcom.com/"
Starting with release 4.5.0 significant changes were introduced to the vco upgrade command "/opt/vc/bin/vco_software_update". The upgrade process is now capable of using signed upgrade packages. This adds an extra layer of security to ensure the integrity of the software being applied.

In order to make use of that additional security layer, please make sure to download the respective public key and place it in the following location. This key should be available to you from the same location where you downloaded the upgrade package.

/var/lib/velocloud/software_update/keys/software.key

You may need to create the respective directory if it doesn't exists:

mkdir /var/lib/velocloud/software_update/keys

The upgrade command provides options should you wish to bypass the signed package verification (–untrusted) or define a different location for the public key to use (--pubkey)

root@vco1:~# /opt/vc/bin/vco_software_update --help

Software Upgrade

Usage:

/opt/vc/bin/vco_software_update [--untrusted] [--batch] [--debug] [--pubkey=KEY]

Copy the VCO upgrade image to: /var/lib/velocloud/software_update/vco_update.tar

cp <image location>/image_name.tar /var/lib/velocloud/software_update/vco_update.tar

VCO upgrade steps :-

1) Upgrade standby VCO (It’s important that this one is done first):

1.1 Execute this step if you upgrading from anything lower than 5.2.2.0 to a version equal or higher than 5.2.2.0. If not, proceed to step 4.1.2

The following MySQL command must be run in the Standby VCO prior to performing the upgrade

ALTER TABLE velocloud_dr.VELOCLOUD_DISASTER_RECOVERY_CONFIG modify column drState ENUM('UNCONFIGURED','ACTIVE_CONFIGURING','ACTIVE_CONFIGURED','ACTIVE_UNCONFIG','LAUNCHING_STANDBY','LAUNCHED_STANDBY','ACTIVE_WAIT_STANDBY','PENDING_STANDBY_CANDIDATE','STANDBY_CANDIDATE','STANDBY_CONFIG_RQST','STANDBY_CONFIGURING','STANDBY_CONFIGURED','PENDING_STANDBY_UNCONFIG','STANDBY_UNCONFIG','PENDING_STANDBY_PROMOTION','STANDBY_PROMOTED','PENDING_ACTIVE_DEMOTION','COPYING_DB','COPY_DB_DONE','COPYING_FILES','COPY_FILES_DONE','SYNC_CONFIGURING','STANDBY_SYNC','STANDBY_BACKGROUND_IMPORT','STANDBY_BACKGROUND_IMPORT_DONE','STANDBY_DATA_MIGRATION','STANDBY_DATA_MIGRATION_IN_PROGRESS','FAILURE_DATA_MIGRATION','STANDBY_DATA_MIGRATION_DONE','STANDBY_RUNNING','UPGRADING','FAILURE_ACTIVE_CONFIGURING','FAILURE_LAUNCHING_STANDBY','FAILURE_STANDBY_CONFIGURING','FAILURE_GET_STANDBY_CONFIG','FAILURE_COPYING_DB','FAILURE_COPYING_FILES','FAILURE_SYNC_CONFIGURING','FAILURE_SYNCING_FILES','FAILURE_GET_STANDBY_STATUS','FAILURE_GET_ACTIVE_STATUS','FAILURE_MYSQL_ACTIVE_STATUS','FAILURE_MYSQL_STANDBY_STATUS','FAILURE_STANDBY_CANDIDATE','FAILURE_STANDBY_UNCONFIG','FAILURE_STANDBY_PROMOTION','FAILURE_ACTIVE_DEMOTION','FAILURE_BACKGROUND_IMPORT','FAILURE_CLICKHOUSE_RSYNC','RECOVERED_CLICKHOUSE_RSYNC')NOT NULL DEFAULT 'UNCONFIGURED';

1.2 Execute the following command as root user to trigger the upgrade process

apt clean all

/opt/vc/bin/vco_software_update

1.3 Reboot.

The upgrade mechanism will determine if this is required and will ask for it to be performed. The user can chose to perform it immediately (advised) or at a later time . The following message appears:

Reboot is required. Reboot? (y/n) [y]

2) Upgrade Primary VCO

2.1 Execute the following command as root user to trigger the upgrade process

apt clean all

/opt/vc/bin/vco_software_update

2.2 Reboot

The upgrade mechanism will determine if this is required and will ask for it to be performed. The user can chose to perform it immediately (advised) or at a later time . The following message appears:

Reboot is required. Reboot? (y/n) [y]

Do not proceed to perform the step below until the reboot (if required) is completed

3) Run schema update

Execute the following script present on the VCO inside the path: /opt/vc/scripts/ with option -e

/opt/vc/scripts/vco_schema_update.sh -e

Other available options for this script

#/opt/vc/scripts/vco_schema_update.sh -h

Usage: vco_schema_update.sh [OPTIONS]

Options:

-h, --help Print this help

-d, --dry-run Print update schema query that is going to execute and exit.

-e, --execute Printe update schema query and execute it.

Actual query execution will be carried out by MySQL event scheduler starting in 60 seconds after executing the query

This script would return immidiately after the execution and delete the schema update sql file

-s, --status Fetch and print the status of the ongoing scheduler execution

If desired, use -d options to see what changes are going to be applied.
To get the current status of the schema changes progress, execute script with -s options and it will print out the current status of schedule schema change. It will also show any ongoing schema changes and progress for it using performance schema. MySQL Documentation with sample

# /opt/vc/scripts/vco_schema_update.sh -s

*************************** 1. row ***************************

Db: velocloud

Name: longRunningOperations

Definer: velocloud-root@localhost

Time zone: SYSTEM

Type: ONE TIME

Execute at: 2022-05-12 07:30:56

Interval value: NULL

Interval field: NULL

Starts: NULL

Ends: NULL

Status: DISABLED

Originator: 1

character_set_client: utf8mb4

collation_connection: utf8mb4_0900_ai_ci

Database Collation: utf8mb4_0900_ai_ci

In case if we see below output after running schema update command, then it means that there are no pending schema changes and it is normal for some VCO upgrades.

root@IN-36-5S-vco:~# 
root@IN-36-5S-vco:~# sudo /opt/vc/scripts/vco_schema_update.sh -e
Schema update file /var/lib/velocloud/file_store/upgradeSchema.sql does not exist. There are no pending schema changes.
root@IN-36-5S-vco:~#

Additional Information

Known Issues:

Insufficient disk space when upgrading SD-WAN orchestrator

When upgrading SD-WAN orchestrator, customer may encounter a issue while trying to perform an upgrade due to insufficient disk space (Bug ID 65998) . If this happens the upgrade will fail to be executed (no harm is done to the running system). Customer will see the following error ending the upgrade command execution:

OSError: [Errno 28] No space left on device

To address this customer may need to create a symbolic link to move the software_update directory out of /var. To do that, simply run the following script:

#!/bin/bash
if [[ ! -L "/var/lib/velocloud/software_update" ]]
then
 mkdir -p /store2/velocloud
 if [[ -d "/var/lib/velocloud/software_update" ]]
 then
 mv /var/lib/velocloud/software_update /store2/velocloud/
 fi
 ln -s /store2/velocloud/software_update /var/lib/velocloud/software_update
fi

This will move the directory to /store2, which is a filesystem that should have plenty of space to manage this extra usage. Once execute this script, customer should be able to proceed with the upgrade process.