High processing Lag and Grid Usage is seen on Aria Operations for Networks version 6.12
Symptoms:
1. GUI Screenshots
2. Data bus is configured with 2 subscriber IDs one with HTTP and another with HTTPs. It is recommended to use HTTPS
3. Flink log exception is seen in log location:
/var/log/hadoop-yarn/containers/application_1722602791013_0001/container_1722602791013_0001_01_000002
Complete exceptions as below:
2024-07-30T05:17:26.153Z ERROR client.gateway.BackPressureFilter BackPressureFilter-scheduled processQueuedRequests:254 Error dispatching requests from the queue java.lang.NoSuchMethodError: 'void com.esotericsoftware.kryo.Kryo_DefaultInstantiatorStrategy.<init>(org.objenesis.strategy.InstantiatorStrategy)'
at com.vmware.lemans.commons.base.serialization.KryoSerializers.create(KryoSerializers.java:61)
at com.vmware.lemans.commons.base.serialization.KryoSerializers_KryoForDocumentThreadLocal.initialValue(KryoSerializers.java:249)
at com.vmware.lemans.commons.base.serialization.KryoSerializers_KryoForDocumentThreadLocal.initialValue(KryoSerializers.java:244)
at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
at com.vmware.lemans.commons.base.serialization.KryoSerializers.getKryoThreadLocalForDocuments(KryoSerializers.java:145)
at com.vmware.lemans.commons.base.serialization.KryoSerializers.clone(KryoSerializers.java:188)
at com.vmware.lemans.commons.base.utils.Utils.clone(Utils.java:92)
at com.vmware.lemans.client.gateway.GatewayOperation.setBody(GatewayOperation.java:53)
at com.vmware.lemans.client.gateway.BackPressureFilter.buildGatewayRequest(BackPressureFilter.java:359)
at com.vmware.lemans.client.gateway.BackPressureFilter.lambda_dispatchRequests_6(BackPressureFilter.java:288)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at com.vmware.lemans.client.gateway.BackPressureFilter.dispatchRequests(BackPressureFilter.java:287)
at com.vmware.lemans.client.gateway.BackPressureFilter.processQueuedRequests(BackPressureFilter.java:227)
at java.base/java.util.concurrent.Executors_RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor_ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor_Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Aria Operations for Networks 6.12.0
Aria Operations for Networks 6.12.1
Kafka's Topic3 partitions were not in sync and hence flink module was not able to read/write properly.
This issue is fixed in Aria Operations for Networks version 6.13
For now we have a workaround to replace the Jar file.
Contact Broadcom Support by opening support case to obtain assistance with correctly replacing the Jar file