“Recent data may not be available due to processing lag of X hour(s)” message in Aria Operations for Network (formerly vRNI)

Products

• VMware Aria Operations for Networks

Issue/Introduction

Symptoms:

• After upgrading or applying patch on Aria Operations for Network, you see this message in the UI console:
  
  Recent data may not be available due to processing lag of X hour(s)
  
  
  
• In the samza-container-0.log file on the platform server, you see entry similar to:
  
  /var/log/hadoop-yarn/containers/application__1489508567362_0001/container__1489508567362_0001_01_000001 (Please Note: the directory numbers for application and container directories will vary depending on the deployment)
  
  Note: Log in to the Platform server as support and use the sudo -i command to change to root context. To find the latest container directories use the ls -la command.
  
  ==> container_1478027833008_0013_01_000002/samza-container-0.log <==
  INFO [2017-06-09 03:04:32,103] [U:687,F:2,090,T:2,776,M:2,842] hbase.client.AsyncProcess:[AsyncProcess$AsyncRequestFutureImpl:resubmit:1223] - [hbase-mutators-36] - #4, table=metrics, attempt=18/35 failed=2676ops, last exception: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region metrics,,1476859388039.c2d3a349b4f6d100411cab36477a182c. is not online on localhost,60020,1496698335187
  
  at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2922)
  at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1053)
  at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2102)
  at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
  at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
  at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
  at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
  at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
  at java.lang.Thread.run(Thread.java:745)
  
  on localhost,60020,1496698335187, tracking started null, retrying after=20154ms, replay=2676ops

Environment

VMware vRealize Network Insight
VMware Aria Operations for Networks

Resolution

This is a known issue affecting Aria Operations for Networks.

Currently, there is no resolution.

To work around this issue, restart the hbase region server:

1. Connect to the Platform virtual machine using SSH and login as support.
2. Run this command to stop the hbase region server:
   
   sudo stop hbase-regionserver
   
   
3. Run this command to start the hbase region server:
   
   sudo start hbase-regionserver
   
   Note: Validate the samza log file to ensure that same occur is not occurring. Messages start to process and still take time to complete.

For related information on changing the support user password, see the modify-password section in  Aria Operations for Networks Command Line Interface User Guide