CWS service flapping after upgrade

Products

VMware VeloCloud SD-WAN with Premier Support VMware SD-WAN by VeloCloud VMware VeloCloud SD-WAN VMWare SD-WAN

Issue/Introduction

After upgrading VCO, the error message “UPGRADE - DEBUG - Waiting for the CWS service up...” kept appearing for about 30 minutes. Finally, upgrading process will be finished with displaying the error message "UPGRADE - ERROR - installer failed with return code 124. See /var/log/vco_software_update.log for details".

It looks that upgrade does not finish correctly.

Cause

The cause of this issue is due to the expiration of the CWS API token.

Resolution

Note: At first, please check "dpkg -l | grep vco". If old vco version is not displayed in the result. the upgrade has been completed successfully and there is only problem by the expiration of the CWS API token.

Workaround

Follow steps 1 through 4 below to update the CWS API token.

1: update the flag

mysql --defaults-extra-file=/etc/mysql/debian.cnf velocloud -e 'update set isActivated = 0 from velocloud.VELOCLOUD_SERVICE_ACCOUNT where username = "cws"'

2: make sure the key in file same as the key in DB

mysql --defaults-extra-file=/etc/mysql/debian.cnf velocloud -e 'select * from velocloud.VELOCLOUD_SERVICE_ACCOUNT where username = "cws"'

*you will see the activation key in the result, it should same as the key in the file /etc/service_accounts/cws_activation_conf.yml

3: delete the existing key

mysql --defaults-extra-file=/etc/mysql/debian.cnf velocloud -e 'delete from velocloud_cws.CWS_VCO_CREDENTIAL'

4: then restart cws

systemctl restart cws-service

Additional Information

To verify that cws-service is running correctly, use the following command. Pleae make sure that “Active: active (running)” is displayed.

service cws-service-ui status

● cws-service-ui.service - VCO CWS service-ui Service
     Loaded: loaded (/lib/systemd/system/cws-service-ui.service; static)
     Active: active (running) since Wed 2024-08-21 05:11:23 UTC; 6 days ago
…

service cws-service status

● cws-service.service - VCO CWS Service
     Loaded: loaded (/lib/systemd/system/cws-service.service; static)
     Active: active (running) since Wed 2024-08-21 05:12:02 UTC; 6 days ago
…

To check the expiration date of the API token in advance, use the following command

mysql --defaults-extra-file=/etc/mysql/debian.cnf velocloud -e 'select * from VELOCLOUD_API_TOKEN'

id      tokenUuid       salt    description     name    created modified        expiration      downloaded      isRevoked       createdByEnterpriseUserId       createdByOperatorUserId createdByServiceAccountId     networkId       enterpriseProxyId       enterpriseId
…
6       aea1b4c2-a752-467b-91a0-7ae4985fc751    zpAARxfmP5y70CaAUswk0m8pugDmAJ6r        NULL    cws33af52509eeb69291176440e58869e4381a2d947466aef8363df1d7ae6347f0e     2024-08-21 05:12:07     2024-08-21 05:12:07   2025-02-19 17:12:07     2024-08-21 05:12:07     0       NULL    NULL    1       1       NULL    NULL
…