Flow-based application discovery fails with "An unexpected error occurred" on VCF Operations for Networks GUI
search cancel

Flow-based application discovery fails with "An unexpected error occurred" on VCF Operations for Networks GUI

book

Article ID: 446115

calendar_today

Updated On:

Products

VCF Operations for Networks

Issue/Introduction

  • Attempt too discover  Flow-based application discovery fails with  "An unexpected error occurred" on VCF Operations for Networks GUI.

    Refer to GUI error message as below:


  • High Processing Lags is seen on the GUI with System Health showing Poor.

    Refer to GUI error message as below:


  • Following error/exception while processing the discovered application are seen from logs at location /var/log/arkin/batch-analytics/batch-analytics.STDOUT-YYYY-MM-DD-HH.MM.SS.log

    ERROR vnera.grid.KafkaPublisher BatchAnalytics_Worker-1 writeToKafka:50 Error in writing to Kafkajava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) [kafka-clients-2.0.1.jar:]	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64) [kafka-clients-2.0.1.jar:]	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) [kafka-clients-2.0.1.jar:]	at com.vnera.grid.KafkaPublisher.writeToKafka(KafkaPublisher.java:48) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.KafkaSdmPublisher.writeToKafka(KafkaSdmPublisher.java:115) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.KafkaSdmPublisher.writeSdmToKafka(KafkaSdmPublisher.java:105) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.uploadHandler.UploadHandler.tryWritingToKafka(UploadHandler.java:732) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.uploadHandler.UploadHandler.publishMessagesToKafka(UploadHandler.java:566) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.uploadHandler.UploadHandler.process(UploadHandler.java:391) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.batchanalytics.task.AppGeneratorTask.uploadSDM(AppGeneratorTask.java:174) [batch-analytics-service-dist-0.001-SNAPSHOT.jar:]	at com.vnera.batchanalytics.task.AppGeneratorTask.runTask(AppGeneratorTask.java:151) [batch-analytics-service-dist-0.001-SNAPSHOT.jar:]	at com.vnera.batchanalytics.task.BatchTask.run(BatchTask.java:89) [batch-analytics-service-dist-0.001-SNAPSHOT.jar:]	at com.vnera.batchanalytics.job.BatchJob.runTasks(BatchJob.java:170) [batch-analytics-service-dist-0.001-SNAPSHOT.jar:]	at com.vnera.batchanalytics.job.BatchJob.execute(BatchJob.java:218) [batch-analytics-service-dist-0.001-SNAPSHOT.jar:]	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.3.2.jar:]	at org.quartz.simpl.SimpleThreadPool_WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.3.2.jar:]

     

  • Following similar error/exception are observed logs in the /var/log/arkin/saasservice/saasservice.STDOUT-YYYY-MM-DD-HH.MM.SS.log.eror  

    ERROR vnera.grid.KafkaPublisher dw-637075 - POST /collectortosaasservlet writeToKafka:50 Error in writing to Kafkajava.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) [kafka-clients-2.0.1.jar:]	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64) [kafka-clients-2.0.1.jar:]	at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) [kafka-clients-2.0.1.jar:]	at com.vnera.grid.KafkaPublisher.writeToKafka(KafkaPublisher.java:48) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.KafkaSdmPublisher.writeToKafka(KafkaSdmPublisher.java:115) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.KafkaSdmPublisher.writeSdmToKafka(KafkaSdmPublisher.java:105) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.uploadHandler.UploadHandler.tryWritingToKafka(UploadHandler.java:732) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.grid.uploadHandler.UploadHandler.publishMessagesToKafka(UploadHandler.java:566) [uploadhandler-0.001-SNAPSHOT.jar:]	at com.vnera.SaasListener.ServiceThriftListener_ServiceImpl.sendMessageToGrid(ServiceThriftListener.java:6185) [saasservice-0.001-SNAPSHOT.jar:]	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at org.eclipse.jetty.util.thread.ReservedThreadExecutor_ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at org.eclipse.jetty.util.thread.QueuedThreadPool_Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.58.v20250814.jar:9.4.58.v20250814]	at java.lang.Thread.run(Thread.java:840) [:]


Environment

VCF Operations for Networks 6.14.2

Cause

  • It identified as a failure or hang in the Kafka service and associated batch-analytics processes. This service disruption prevented the system from processing flow data required for application discovery.
  • Kafka is responsible for message queuing and flow metric processing. When the Kafka cluster or individual services (like batch-analytics.service) become unavailable or lose synchronization, flow data cannot be ingested.
  • This results in Flow-based application discovery fails with  "An unexpected error occurred" on VCF Operations for Networks GUI.

Resolution

VCF Operations for Networks is aware of this issue and current no fix is available.

To workaround this issue, perform below:

  1. Validate if Zookeeper is reachable.
  2. Open a SSH session to any platform node1 and switch to ubuntu user.
    Execute below two commands:
    ub
/home/ubuntu/run_all.sh 'echo ruok | nc.traditional -w 1 localhost 2181 | grep imok'
/home/ubuntu/run_all.sh 'echo isro | nc.traditional -w 1 localhost 2181 | grep rw'
  3. You should see below response to confirm Zookeeper is reachable:
    support@aria-networks-platform1:~$ ub
ubuntu@aria-networks-platform1:~$ /home/ubuntu/run_all.sh 'echo ruok | nc.traditional -w 1 localhost 2181 | grep imok'
imok
ubuntu@aria-networks-platform1:~$ /home/ubuntu/run_all.sh 'echo isro | nc.traditional -w 1 localhost 2181 | grep rw'
rw
ubuntu@aria-networks-platform1:~$

     

  4. Execute below commands on platform node1:
    /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic Topic3

  5.  

    Output of above commands should show as below:
    ubuntu@platform1:~$ /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic Topic3
Topic: Topic3   PartitionCount: 30      ReplicationFactor: 3    Configs: min.insync.replicas=2,segment.bytes=536870912,max.message.bytes=15000000,unclean.leader.election.enable=false,retention.bytes=5368709120
        Topic: Topic3   Partition: 0    Leader: 0       Replicas: 0,4,5 Isr: 0,5,4
        Topic: Topic3   Partition: 1    Leader: 2       Replicas: 2,5,0 Isr: 0,2,5
        Topic: Topic3   Partition: 2    Leader: 3       Replicas: 3,0,2 Isr: 3,0,2
        Topic: Topic3   Partition: 3    Leader: 4       Replicas: 4,2,3 Isr: 3,2,4
        Topic: Topic3   Partition: 4    Leader: 5       Replicas: 5,3,4 Isr: 3,5,4
        Topic: Topic3   Partition: 5    Leader: 0       Replicas: 0,5,2 Isr: 0,2,5
        Topic: Topic3   Partition: 6    Leader: 2       Replicas: 2,0,3 Isr: 0,3,2
        Topic: Topic3   Partition: 7    Leader: 3       Replicas: 3,2,4 Isr: 3,2,4
        Topic: Topic3   Partition: 8    Leader: 4       Replicas: 4,3,5 Isr: 3,5,4
        Topic: Topic3   Partition: 9    Leader: 5       Replicas: 5,4,0 Isr: 0,5,4
        Topic: Topic3   Partition: 10   Leader: 0       Replicas: 0,2,3 Isr: 0,3
        Topic: Topic3   Partition: 11   Leader: 2       Replicas: 2,3,4 Isr: 3,2,4
        Topic: Topic3   Partition: 12   Leader: 3       Replicas: 3,4,5 Isr: 3,5,4
        Topic: Topic3   Partition: 13   Leader: 4       Replicas: 4,5,0 Isr: 0,5,4
        Topic: Topic3   Partition: 14   Leader: 5       Replicas: 5,0,2 Isr: 0,2,5
        Topic: Topic3   Partition: 15   Leader: 0       Replicas: 0,3,4 Isr: 0,3,4
        Topic: Topic3   Partition: 16   Leader: 2       Replicas: 2,4,5 Isr: 4,5,2
        Topic: Topic3   Partition: 17   Leader: 3       Replicas: 3,5,0 Isr: 0,3,5
        Topic: Topic3   Partition: 18   Leader: 4       Replicas: 4,0,2 Isr: 0,2,4
        Topic: Topic3   Partition: 19   Leader: 5       Replicas: 5,2,3 Isr: 3,2,5
        Topic: Topic3   Partition: 20   Leader: 0       Replicas: 0,4,5 Isr: 0
        Topic: Topic3   Partition: 21   Leader: 2       Replicas: 2,5,0 Isr: 0,2,5
        Topic: Topic3   Partition: 22   Leader: 3       Replicas: 3,0,2 Isr: 0,3,2
        Topic: Topic3   Partition: 23   Leader: 4       Replicas: 4,2,3 Isr: 3,2,4
        Topic: Topic3   Partition: 24   Leader: 5       Replicas: 5,3,4 Isr: 3,5,4
        Topic: Topic3   Partition: 25   Leader: 0       Replicas: 0,5,2 Isr: 0,2,5
        Topic: Topic3   Partition: 26   Leader: 2       Replicas: 2,0,3 Isr: 0,3,2
        Topic: Topic3   Partition: 27   Leader: 3       Replicas: 3,2,4 Isr: 3,2,4
        Topic: Topic3   Partition: 28   Leader: 4       Replicas: 4,3,5 Isr: 3,5,4
        Topic: Topic3   Partition: 29   Leader: 5       Replicas: 5,4,0 Isr: 0,5,4
    In above output we see that the partition #20 is a  case of Insufficient replicas. ISR=1 and min.insyc.replicas=2, isr < min.insyc.replicas.
    When this situation occurs kafka don't accept any messages for that Topic.

  6. Stop the following services in order as per given commands below:
    /home/ubuntu/run_all.sh sudo systemctl stop flinkjobs.service
/home/ubuntu/run_all.sh sudo systemctl stop kafka.service
/home/ubuntu/run_all.sh sudo systemctl stop zookeeper-server.service

     

  7. Start the following services in order as per given commands below:
    /home/ubuntu/run_all.sh sudo systemctl start zookeeper-server.service
/home/ubuntu/run_all.sh sudo systemctl start kafka.service
/home/ubuntu/run_all.sh sudo systemctl start flinkjobs.service


  8. Re-capture the output of below command:
     /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic Topic3
  9. Output of above commands should show as below:
    ubuntu@platform1:~$ /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic Topic3
Topic: Topic3   PartitionCount: 30      ReplicationFactor: 3    Configs: min.insync.replicas=2,segment.bytes=536870912,max.message.bytes=15000000,unclean.leader.election.enable=false,retention.bytes=5368709120
        Topic: Topic3   Partition: 0    Leader: 0       Replicas: 0,4,5 Isr: 0,5,4
        Topic: Topic3   Partition: 1    Leader: 2       Replicas: 2,5,0 Isr: 0,2,5
        Topic: Topic3   Partition: 2    Leader: 3       Replicas: 3,0,2 Isr: 3,0,2
        Topic: Topic3   Partition: 3    Leader: 4       Replicas: 4,2,3 Isr: 3,2,4
        Topic: Topic3   Partition: 4    Leader: 5       Replicas: 5,3,4 Isr: 3,5,4
        Topic: Topic3   Partition: 5    Leader: 0       Replicas: 0,5,2 Isr: 0,2,5
        Topic: Topic3   Partition: 6    Leader: 2       Replicas: 2,0,3 Isr: 0,3,2
        Topic: Topic3   Partition: 7    Leader: 3       Replicas: 3,2,4 Isr: 3,2,4
        Topic: Topic3   Partition: 8    Leader: 4       Replicas: 4,3,5 Isr: 3,5,4
        Topic: Topic3   Partition: 9    Leader: 5       Replicas: 5,4,0 Isr: 0,5,4
        Topic: Topic3   Partition: 10   Leader: 0       Replicas: 0,2,3 Isr: 0,3
        Topic: Topic3   Partition: 11   Leader: 2       Replicas: 2,3,4 Isr: 3,2,4
        Topic: Topic3   Partition: 12   Leader: 3       Replicas: 3,4,5 Isr: 3,5,4
        Topic: Topic3   Partition: 13   Leader: 4       Replicas: 4,5,0 Isr: 0,5,4
        Topic: Topic3   Partition: 14   Leader: 5       Replicas: 5,0,2 Isr: 0,2,5
        Topic: Topic3   Partition: 15   Leader: 0       Replicas: 0,3,4 Isr: 0,3,4
        Topic: Topic3   Partition: 16   Leader: 2       Replicas: 2,4,5 Isr: 4,5,2
        Topic: Topic3   Partition: 17   Leader: 3       Replicas: 3,5,0 Isr: 0,3,5
        Topic: Topic3   Partition: 18   Leader: 4       Replicas: 4,0,2 Isr: 0,2,4
        Topic: Topic3   Partition: 19   Leader: 5       Replicas: 5,2,3 Isr: 3,2,5
        Topic: Topic3   Partition: 20   Leader: 0       Replicas: 0,4,5 Isr: 0,2,4
        Topic: Topic3   Partition: 21   Leader: 2       Replicas: 2,5,0 Isr: 0,2,5
        Topic: Topic3   Partition: 22   Leader: 3       Replicas: 3,0,2 Isr: 0,3,2
        Topic: Topic3   Partition: 23   Leader: 4       Replicas: 4,2,3 Isr: 3,2,4
        Topic: Topic3   Partition: 24   Leader: 5       Replicas: 5,3,4 Isr: 3,5,4
        Topic: Topic3   Partition: 25   Leader: 0       Replicas: 0,5,2 Isr: 0,2,5
        Topic: Topic3   Partition: 26   Leader: 2       Replicas: 2,0,3 Isr: 0,3,2
        Topic: Topic3   Partition: 27   Leader: 3       Replicas: 3,2,4 Isr: 3,2,4
        Topic: Topic3   Partition: 28   Leader: 4       Replicas: 4,3,5 Isr: 3,5,4
        Topic: Topic3   Partition: 29   Leader: 5       Replicas: 5,4,0 Isr: 0,5,4
  10. Validate that all services are running an healthy using below command:
    ./run_all.sh sudo /home/ubuntu/check-service-health.sh -p -d
  11.  Navigate back to GUI and re trigger Flow-based application discovery

  12. Flow Based application discovery should be processed and error shouldn't be seen, Application Discovery Feature should work as expected.
Powered by Wolken Software