NF Instantiation Fails with 'No resources found' in Namespace due to Kafka RecordTooLargeException

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

When attempting to instantiate a Network Function (NF) in VMware Telco Cloud Automation (TCA), the process fails with the following symptoms:

• The TCA UI reports an error: "No resources found in [namespace]".
  
  
• Subsequent Life Cycle Management (LCM) activities, such as Reconfigure or Scale, are disabled or fail.
  
  
• In the TCA-CP edge-kafka-0 pod logs, the following exception is observed: org.apache.kafka.common.errors.RecordTooLargeException: The message is 2582931 bytes consumed by the partition [topic-name] and it may not exceed 2097152 bytes.

Environment

TCA 3.2
TCP 5.0

Cause

This issue is caused by a Kafka message size limitation. By default, Kafka is configured with a max.request.size of 2,097,152 bytes (2MB).

During NF instantiation, the workflow generates a serialized job data payload. If this payload (often containing large configuration files, logs, or status updates) exceeds 2MB, Kafka rejects the message. This prevents the TCA Control Plane (TCA-CP) from synchronizing the job state with the TCA Manager (TCA-M), leading to the "No resources found" state.

Resolution

To resolve this issue, you must purge the stuck Kafka topics and ensure workflow outputs are optimized.

Note: Please make sure snapshot is taken before applying these changes so that we can goto the previously configured state later if required.

Purge Kafka Topics on TCA-CP

1. Log in to the TCA-CP via SSH.
2. Pause TCA App Reconciliation:
   

   kubectl patch app tca -n tca-cp-cn --type='merge' -p '{"spec": {"paused": true}}'

3. Then Delete the api and app pods
4. Go to kafka pod terminal
   

   kubectl exec -ti edge-kafka-0 -n tca-cp-cn -- bash

5. Change to bin directory and Get current size of kafka logs (db):
   

   du -sh /var/lib/kafka/data/*5.3G    /var/lib/kafka/data/kafka-log0

6. Change to the correct directory before running the scripts
   

   cd /opt/kafka/bin

7. This set the retention time of topics to 0ms (purging topics):
   

   for t in $(./kafka-topics.sh --bootstrap-server localhost:9092 --list); do  ./kafka-configs.sh --bootstrap-server localhost:9092   --alter --entity-type topics --entity-name $t  --add-config retention.ms=0;done    

8. Eventually the size will reduce to 82MB from 5.5GB
   

   du -sh /var/lib/kafka/data/*82M    /var/lib/kafka/data/kafka-log0

9. Revert Retention Settings: Wait a few minutes for the logs to clear, then revert the retention config:
   

   for t in $(./kafka-topics.sh --bootstrap-server localhost:9092 --list); do   ./kafka-configs.sh --bootstrap-server localhost:9092   --alter --entity-type topics --entity-name $t   --delete-config retention.ms; done
   

10. Exit Pod and Resume Reconciliation:
    

    kubectl patch app tca -n tca-cp-cn --type='merge' -p '{"spec": {"paused": false}}'

    Once the above workaround is applied, perform NF instantiation