vRealize Network Insight Upgrade from 6.0 to 6.1 Fails on ES Migration Step

Products

• VMware Aria Operations for Networks

Issue/Introduction

Symptoms:

• The migration process appears to be stuck during the upgrade.
• The UI dialogue will show no progress.

Environment

VMware vRealize Network Insight 6.x

Cause

• Elasticsearch (ES) scans objects in batches with a predefined memory
• In case of a large size of batches, ES runs out of memory

Resolution

Problem Identification:

Run the command: 
grep OutOfMemoryError /var/log/arkin/esreindexer_workflow.log

If the command returns any output, proceed to the steps in the Workaround below.

Workaround:

1. Reduce scanBatchSize parameter from 50 to 10 in the following file on the platform nodes where reindexing failed.

/home/ubuntu/build-target/common-utils
 

Note: Reducing scanBatchSize parameter reduces the amount of heap memory required to process the objects and reduces heap memory consumption and prevents OOM errors. This change also reduces the reindexing rate.

2. Stop launcher service on all the nodes

/home/ubuntu/run_all.sh sudo systemctl stop launcher-service

 

3. Perform a grep for reindexer.sh script and kill the processes on all the platform nodes

 ps -ef | grep reindexer.sh
 kill -9 <process ids returned by above command>
 

4. Start launcher service on all the nodes

/home/ubuntu/run_all.sh sudo systemctl start launcher-service