VCO upgrade steps to upgrade VCO from from any version starting from 5.x to 6.x.

Products

VMWare SD-WAN VMware SD-WAN by VeloCloud

Issue/Introduction

This document should be followed only for upgrades to 6.x versions.

Environment

VMware SDWAN Orchestrator

Cause

NA

Resolution

Pre-requisites before starting VCO upgrade :-

Prepare for VCO Upgrade (Follow the VMware SASE Orchestrator Deployment and Monitoring Guide)
Send upgrade announcement from Orchestrator Upgrade screen on VCO

All commands presented here are intended to be executed by "root"
While the process can be executed over SSH (and might be the only option in certain environments), it is recommended to be done from the VM console to avoid potential issues that might happen if the SSH connection is interrupted for any reason.
You may obtain software images from "https://support.broadcom.com/"
Starting with release 4.5.0 significant changes were introduced to the vco upgrade command "/opt/vc/bin/vco_software_update". The upgrade process is now capable of using signed upgrade packages. This adds an extra layer of security to ensure the integrity of the software being applied.

In order to make use of that additional security layer, please make sure to download the respective public key and place it in the following location. This key should be available to you from the same location where you downloaded the upgrade package.

/var/lib/velocloud/software_update/keys/software.key

You may need to create the respective directory if it doesn't exists:

mkdir /var/lib/velocloud/software_update/keys

The upgrade command provides options should you wish to bypass the signed package verification (–untrusted) or define a different location for the public key to use (--pubkey)

root@vco1:~# /opt/vc/bin/vco_software_update --help

Software Upgrade

Usage:

/opt/vc/bin/vco_software_update [--untrusted] [--batch] [--debug] [--pubkey=KEY]

Copy the VCO upgrade image to: /var/lib/velocloud/software_update/vco_update.tar

cp <image location>/image_name.tar /var/lib/velocloud/software_update/vco_update.tar

VCO upgrade steps :-

1) Upgrade standby VCO (It’s important that this one is done first):

1.0 Ensure that there is atleast 30 GB of VFree for 'vols' disk (This is only required for upgrades to 6.x VCO versions)
As a prerequisite for the upgrade, the 'vols' volume group should have atleast 30 GB of free space (It is indicated by VFree).

sudo vgs
  VG     #PV #LV #SN Attr   VSize    VFree
  vols     1   7   0 wz--n- <127.90g   40.68g

1.1 Execute this step if you upgrading from anything lower than 5.2.2.0 to a version equal or higher than 5.2.2.0. If not, proceed to step 4.1.2

The following MySQL command must be run in the Standby VCO prior to performing the upgrade

ALTER TABLE velocloud_dr.VELOCLOUD_DISASTER_RECOVERY_CONFIG modify column drState ENUM('UNCONFIGURED','ACTIVE_CONFIGURING','ACTIVE_CONFIGURED','ACTIVE_UNCONFIG','LAUNCHING_STANDBY','LAUNCHED_STANDBY','ACTIVE_WAIT_STANDBY','PENDING_STANDBY_CANDIDATE','STANDBY_CANDIDATE','STANDBY_CONFIG_RQST','STANDBY_CONFIGURING','STANDBY_CONFIGURED','PENDING_STANDBY_UNCONFIG','STANDBY_UNCONFIG','PENDING_STANDBY_PROMOTION','STANDBY_PROMOTED','PENDING_ACTIVE_DEMOTION','COPYING_DB','COPY_DB_DONE','COPYING_FILES','COPY_FILES_DONE','SYNC_CONFIGURING','STANDBY_SYNC','STANDBY_BACKGROUND_IMPORT','STANDBY_BACKGROUND_IMPORT_DONE','STANDBY_DATA_MIGRATION','STANDBY_DATA_MIGRATION_IN_PROGRESS','FAILURE_DATA_MIGRATION','STANDBY_DATA_MIGRATION_DONE','STANDBY_RUNNING','UPGRADING','FAILURE_ACTIVE_CONFIGURING','FAILURE_LAUNCHING_STANDBY','FAILURE_STANDBY_CONFIGURING','FAILURE_GET_STANDBY_CONFIG','FAILURE_COPYING_DB','FAILURE_COPYING_FILES','FAILURE_SYNC_CONFIGURING','FAILURE_SYNCING_FILES','FAILURE_GET_STANDBY_STATUS','FAILURE_GET_ACTIVE_STATUS','FAILURE_MYSQL_ACTIVE_STATUS','FAILURE_MYSQL_STANDBY_STATUS','FAILURE_STANDBY_CANDIDATE','FAILURE_STANDBY_UNCONFIG','FAILURE_STANDBY_PROMOTION','FAILURE_ACTIVE_DEMOTION','FAILURE_BACKGROUND_IMPORT','FAILURE_CLICKHOUSE_RSYNC','RECOVERED_CLICKHOUSE_RSYNC')NOT NULL DEFAULT 'UNCONFIGURED';

1.2 Execute the following command as root user to trigger the upgrade process

apt clean all

/opt/vc/bin/vco_software_update

1.3 Reboot.

The upgrade mechanism will determine if this is required and will ask for it to be performed. The user can chose to perform it immediately (advised) or at a later time . The following message appears:

Reboot is required. Reboot? (y/n) [y]

2) Upgrade Primary VCO

2.0 Ensure that there is atleast 30 GB of VFree for 'vols' disk (This is only required for upgrades to 6.x VCO versions)
As a prerequisite for the upgrade, the 'vols' volume group should have atleast 30 GB of free space (It is indicated by VFree).

sudo vgs
  VG     #PV #LV #SN Attr   VSize    VFree
  vols     1   7   0 wz--n- <127.90g   40.68g

2.1 Execute the following command as root user to trigger the upgrade process

apt clean all

/opt/vc/bin/vco_software_update

2.2 Reboot

The upgrade mechanism will determine if this is required and will ask for it to be performed. The user can chose to perform it immediately (advised) or at a later time . The following message appears:

Reboot is required. Reboot? (y/n) [y]

Do not proceed to perform the step below until the reboot (if required) is completed

3) Run schema update

Execute the following script present on the VCO inside the path: /opt/vc/scripts/ with option -e

/opt/vc/scripts/vco_schema_update.sh -e

Other available options for this script

#/opt/vc/scripts/vco_schema_update.sh -h

Usage: vco_schema_update.sh [OPTIONS]

Options:

-h, --help Print this help

-d, --dry-run Print update schema query that is going to execute and exit.

-e, --execute Printe update schema query and execute it.

Actual query execution will be carried out by MySQL event scheduler starting in 60 seconds after executing the query

This script would return immidiately after the execution and delete the schema update sql file

-s, --status Fetch and print the status of the ongoing scheduler execution

If desired, use -d options to see what changes are going to be applied.
To get the current status of the schema changes progress, execute script with -s options and it will print out the current status of schedule schema change. It will also show any ongoing schema changes and progress for it using performance schema. MySQL Documentation with sample

# /opt/vc/scripts/vco_schema_update.sh -s

*************************** 1. row ***************************

Db: velocloud

Name: longRunningOperations

Definer: velocloud-root@localhost

Time zone: SYSTEM

Type: ONE TIME

Execute at: 2022-05-12 07:30:56

Interval value: NULL

Interval field: NULL

Starts: NULL

Ends: NULL

Status: DISABLED

Originator: 1

character_set_client: utf8mb4

collation_connection: utf8mb4_0900_ai_ci

Database Collation: utf8mb4_0900_ai_ci

In case if we see below output after running schema update command, then it means that there are no pending schema changes and it is normal for some VCO upgrades.

root@IN-36-5S-vco:~# 
root@IN-36-5S-vco:~# sudo /opt/vc/scripts/vco_schema_update.sh -e
Schema update file /var/lib/velocloud/file_store/upgradeSchema.sql does not exist. There are no pending schema changes.
root@IN-36-5S-vco:~#

Additional Information

For On Prem Setup, if you need assistance with VCO upgrade, kindly engage Broadcom Support.

In case, you need help to increase VFree for "vols", you can follow below steps :-

1) Add Hard disk 5 from VCENTER/ESXi to the VCO VM
2) Run "sudo pvcreate /dev/sde"
3) Run "vgextend vols /dev/sde"