TCA 2.0.1 Read timed out while upgrading Cloud-Native Telco Cloud Automation

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

To provide a workaround in case customer faces read timed out error while upgrading Cloud-Native Telco Cloud Automation

Symptoms:

When customer is upgrading Cloud-Native Telco Cloud Automation. Upgrade can possibly fail throwing read timed out error. Example: "Failure while setting upgrade agent: Helm API failed: Read timed out"

Environment

VMware Telco Cloud Automation 2.0.1
VMware Telco Cloud Automation 2.0

Cause

TCA Service taking longer to respond and send information. Could also be due to slow network connection.

Resolution

This issue is fixed in Telco Cloud Automation 2.1.0

Workaround:
When customer is running VMware Telco Cloud Automation 2.0.0 and 2.0.1 and notices a read timed out error when trying to upgrade tca app-engine pod needs to be patched.

Note: If TCA Manager and TCA CP are running in the same cluster. It is recommended to patch both of them before upgrading. Then upgrade TCA Manager first and upgrade TCA CP after that.

Please flow the steps below:

1. SSH to TCA bootstrapper VM and login as root.

2. Fetch the patched docker image of TCA app-engine as below:

 VERSION = Telco Cloud Automation Version, for example here it is 2.0.0 or 2.0.1
 TCA-BOOTSTRAPPER-IP = IP Address of the TCA Bootstrapper VM.

docker pull vmwaresaas.jfrog.io/registry/app-engine:{VERSION}-patch

docker tag vmwaresaas.jfrog.io/registry/app-engine:{VERSION}-patch {TCA-BOOTSTRAPPER-IP}:8012/library/app-engine:{VERSION}

docker push {TCA-BOOTSTRAPPER-IP}:8012/library/app-engine:{VERSION}

3. Patching app-engine in TCA-Manager is as below:
 
  helm get values tca -n tca-mgr > tca-mgr.yaml

  NOTE: If you are unable to connect to TCA Mgmt cluster or if tca-mgr.yaml is empty. Please set KUBECONFIG variable by running 'export KUBECONFIG=/opt/vmware/k8s-bootstrapper/b7ac1837-1b85-4522-a715-6cd577029ec5/kubeconfig' b7ac1837-1b85-4522-a715-6cd577029ec5 is a unique id, it will be different for every TCA Mgmt cluster.

  Open tca-mgr.yaml and set

image.pullPolicy: Always

  Example:

  [root@bootstrapperVM36 ~]# cat tca-mgr.yaml
  USER-SUPPLIED VALUES:
  appliance:
    role: tca-manager
  image:
    repository: 10.176.113.36:8012/library
    tcaCommonTag: 2.0.1
    pullPolicy: Always ## This needs to be added.
  ingress:
    enabled: true

 After saving tca-mgr.yaml, run 'helm upgrade tca chartmuseum/library/tca -f tca-mgr.yaml -n tca-mgr'. This will patch app-engine pod. Once TCA services are up in few minutes you can retry the TCA-Mgr upgrade.

4. Patching app-engine in TCA-CP is as below:
   

helm get values tca -n tca-system > tca-cp.yaml

  NOTE: If you are unable to connect to TCA Mgmt cluster or if tca-cp.yaml is empty. Please set KUBECONFIG variable by running 'export KUBECONFIG=/opt/vmware/k8s-bootstrapper/b7ac1837-1b85-4522-a715-6cd577029ec5/kubeconfig' b7ac1837-1b85-4522-a715-6cd577029ec5 is a unique id, it will be different for every TCA Mgmt cluster.

  Open tca-cp.yaml and set image.pullPolicy: Always

  Example:
 

 [root@bootstrapperVM36 ~]# cat tca-cp.yaml
  USER-SUPPLIED VALUES:
  appliance:
    role: tca-cp
  image:
    repository: 10.176.113.36:8012/library
    tcaCommonTag: 2.0.1
    pullPolicy: Always ## This needs to be added.
  ingress:
    enabled: true

 Post saving tca-cp.yaml, run 'helm upgrade tca chartmuseum/library/tca -f tca-cp.yaml -n tca-system'. This will patch app-engine pod. Once TCA services are up in few minutes you can retry the TCA-CP upgrade.