Nodepolicy creation failure due to kafka issue

Products

• VMware Telco Cloud Automation

Issue/Introduction

Below error was observed in the machine nodepolicy.

  reason: >-
    vmconfig status is Normal, nodeconfig status is Normal. vmconfig CR
    wc-tah-test/wc-tah-test-np1-66666ffss4-v11nx does not exist and
    nodepolicy CR wc-tah-test/wc-tah-test-np1 does not exist.

Environment

3.2

Cause

Nodepolicy creation is failing as the vsphereJobs on TCA-CP Compute was not consumed by the tca-app service from kafka which resulted in kafka message lag.

Resolution

Follow below steps to check the issue:

1. Login to the TCA-CP and Check if Service is Keeping-up with Kafka Messages 
   

   admin@dual-tca-cp1 [ ~ ]$ kubectexec -it -n tca-cp-cn edge-kafka-0 /bin/bash
   kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] – [COMMAND] instead.
   kafka [ /opt/kafka ]$ cd bin
   kafka [ /opt/kafka/bin ]$ ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group vsphereJobs-0-TopologyService
   
   Consumer group 'vsphereJobs-0-TopologyService' has no active members.
   
   GROUP                         TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
   vsphereJobs-0-TopologyService vsphereJobs     0          156916          157073          157             -               -               -
   
       

    

2. Verify if there is any LAG observed between CURRENT-OFFSET and LOG-END-OFFSET for kafka-consumer-group vsphereJobs-0-TopologyService. 

   Note: The difference between LOG-END-OFFSET and CURRENT-OFFSET is called LAG which should be zero. If its not zero the service is not consuming messages from the Kafka topic. Issue is likely with Service.

   
   Please collect the TCA support bundle, service/pod logs and open a Broadcom support ticket to troubleshoot this issue further.