Network Functions get stuck when instantiating a CNF instead of timing out.

Products

• VMware Telco Cloud Automation

Issue/Introduction

The purpose of this patch is to fix the memory leak identified and introduce a timeout for operations that are in a stuck state and limit the number of helm-service threads that are active on the Telco Cloud Automation Control Plane (TCA-CP)) at any given moment.

Symptoms:
Network Functions get stuck when instantiating a CNF instead of timing out.

Environment

2.0.x

Cause

In certain scenarios the memory footprint of the helm-service, both RAM and swap, will drastically increase. This will ultimately lead to an out of memory condition.

Resolution

Resolved in Telco Cloud Automation 2.1.

Workaround:
1. Download the attached helm-service binary for your version of TCA.
 
2. SSH into the TCA-CP virtual machine.
 
3. Stop the helm-service
systemctl stop helm-service
 
4. Rename the binary to helm-service and copy the new binary to the following folder:
/opt/vmware/bin/
 
5. Add executable permissions to the binary
chmod 775 /opt/vmware/bin/helm-service
 
6. Start the helm-service
systemctl start helm-service
 
New environment variables:

• HELM_SERVICE_INSTALL_TIMEOUT: Timeout before failing installation jobs. (Default value is 5 minutes).
• HELM_SERVICE_UPGRADE_TIMEOUT: Timeout before failing upgrade jobs. (Default value is 5 minutes).
• HELM_SERVICE_UNINSTALL_TIMEOUT: Timeout before failing uninstall jobs. (Default value is 5 minutes).
• HELM_SERVICE_MAX_ROUTINES: The maximum number of parallel helm-service jobs to execute, additional jobs will be queued until a worker is available. (Default value is 10).
• HELM_SERVICE_WAITGROUP_ENABLED: Turns on/off the parallel worker limit, set to False to allow unlimited workers. (Default value is true).

 
To apply new environment variables:
Add the environment variables by modifying the systemd environment file located in the following folder of the TCA-CP
/etc/hybridity/helm-service.conf