TCA GUI terminal inaccessible and long upgrade delays with large helm value files

Products

• VMware Telco Cloud Automation

Issue/Introduction

When performing a Helm upgrade for a CNF, the following symptoms are observed:

• The upgrade process takes an exceptionally long time to start (sometimes exceeding one hour).
• The TCA Manager (TCA-M) GUI terminal becomes stuck or unresponsive for all tenants and CNFs during the operation.
• The issue is typically associated with CNFs containing a large number of Helm charts or very large values.yaml files.

Environment

3.4

Cause

This issue is caused by the processing overhead of large or numerous values.yaml files. During the upgrade process, TCA base64 encodes these files and includes them in log statements. When a CNF has many Helm charts, the cumulative size of these encoded strings creates significant messaging overhead and resource contention, leading to GUI unresponsiveness and initiation delays.

Resolution

This issue is resolved in TCA 3.4.0.1. Please refer 3.4.0.1 release notes

Workaround

If an immediate upgrade is not possible, the following workaround can alleviate the impact:

Suppress Verbose Logging: Reduce the logging level for CNF operations to minimize the processing of large base64 strings in log outputs.

The following steps must be executed within the TCA-M and all relevant TCA-CPs to effectively stabilize the environment.

1. Access the app pod:  Identify the active tca-app pod and enter the shell:
   

   kubectl exec -it tca-mgr tca-app-<podname>  -- bash

2. Move to config folder
   

   cd config

3.  Inject Logger Suppressions: Execute the sed commands to modify app-logback.xml. These commands insert specific "OFF" level directives for the orchestrator jobs before the root logger definition.
   
   
   • Suppress CNF Reconfigure Logging
     

     sed -i 's#<root level="INFO">#<logger name="com.vmware.hybridity.service.nfv.vim.workflows.orch.CnfReconfigureJob" level="OFF" /><root level="INFO">#g' app-logback.xml

   • Suppress Component Reconfigure Logging
     

     sed -i 's#<root level="INFO">#<logger name="com.vmware.hybridity.service.nfv.vim.workflows.orch.ComponentReconfigureJob" level="OFF" /><root level="INFO">#g' app-logback.xml

   • Suppress CNF VDU Upgrade Logging
     

     sed -i 's#<root level="INFO">#<logger name="com.vmware.hybridity.service.nfv.vim.workflows.orch.CnfVduUpgradeJob" level="OFF" /><root level="INFO">#g' app-logback.xml