VMware vRealize Network Insight Cluster reboot procedure

Products

• VMware Aria Operations for Networks
• VMware Aria Suite

Issue/Introduction

 a. Reboot of N node cluster setup in an appropriate order along with services on each node.

 b. Safe stop and start of services coordinated across cluster nodes to avoid any data corruption.

Symptoms:
In vRNI Platform cluster environment services become unhealthy or data corruption occurs due to improper shutdown or reboot of cluster nodes.

This script has to be run on a Linux/Unix system for it to work.

 

Environment

VMware vRealize Network Insight 6.4.0
VMware vRealize Network Insight 6.3.x
VMware vRealize Network Insight 6.6.0
VMware vRealize Network Insight 6.8.0
VMware vRealize Network Insight 6.2.x
VMware vRealize Network Insight 6.5.1
VMware vRealize Network Insight 6.7.0
VMware vRealize Network Insight 6.9.0
VMware Aria Operations for Networks 6.10.0
VMware Aria Operations for Networks 6.11.0
VMware Aria Operations for Networks 6.12.1

Resolution

Steps to run/execute:
1. Download the script vrni-reboot-script.sh (See Attachment section)
2. Provide execution permissions to  shell script.
3. Execute as ./vrni-reboot-script.sh <restart-services|stop-services|start-services|reboot|shutdown> <PLATFORM_IP> [<LOG_FILE_PATH>]

NOTE: It is expected to run on a non-vRNI host with SSH access to vRNI Platform.
            Password of "support" user will be prompted once and further communication will be based on generated SSH key pair.

Example invocations:

To properly stop the services and shutdown the cluster for taking snapshots/backups:

./vrni-reboot-script.sh stop-services "192.168.0.1"   - To stop all cluster services with only console logging
./vrni-reboot-script.sh shutdown "192.168.0.1" "/home/vRNI_010123_shutdown.log"   - To reboot cluster nodes and save log in specified path

 

Ip address mentioned in above examples 192.168.0.1 is an example IP address, in customer's vRNI cluster deployment this  would be the IP address for platform 1 Node.
The remaining cluster nodes will be shutdown through platform node1.
      
Functionality:
reboot - Restart services and nodes in the cluster in the right order
restart-services - Restart services in cluster in the right order
stop-services - Stop All services in the cluster
start-services - Start All services in the cluster
shutdown - Shutdown all platform nodes in the cluster for taking snapshots etc.

Use Cases:

1. Disk usage is at alarming level and nodes uptime is high.
2. Some external backup is scheduled which require shut down of all nodes.
3. Routine power cycle or reboot cluster nodes due to some business policy.
4. Shutdown/Restarting cluster services (as directed by GSS) to clear up some temporary inconsistencies or for any other reason.

NOTE:
The script works on both IPv4 and IPv6 vRNI cluster setups.

Additional Information

Impact/Risks:
vRNI version 6.2.0 and onwards.