Post upgrading to VMware Aria Operations for Networks On-Premises version 6.11, you may see a 'Launcher Service Not Running' error.

Products

• VMware Aria Operations for Networks
• VMware Aria Suite

Issue/Introduction

Symptoms:

This is seen in simple deployments with 1 Platform node only.
After upgrading to VMware Aria Operations for Networks version 6.11, you may see Launcher Service Not Running error on GUI.

1. Below is  alert/error  is seen on VMware Aria Operations for Networks  GUI.



2. If Alert notifications are configured then below alert notification is generated as well.



3. On Platform node navigate to launcher latest.log file. Log location: /home/ubuntu/logs/launcher
    Search with keyword vrni-platform-release (This is old name) to see exceptions similar to such as below.   
   You see entries similar to below:

2023-09-14T13:43:48.627Z ERROR zookeeper.client.StaticHostProvider ReadOnlyZKClient-vrni-platform-release:2181@0x5c0d876c-SendThread(vrni-platform-release:2181) resolve:154 Unable to resolve address: vrni-platform-release:2181
java.net.UnknownHostException: vrni-platform-release
        at java.net.InetAddress_CachedAddresses.get(InetAddress.java:797) _[_:_]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1519) _[_:_]
        at java.net.InetAddress.getAllByName(InetAddress.java:1378) _[_:_]
        at java.net.InetAddress.getAllByName(InetAddress.java:1306) _[_:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.client.StaticHostProvider_1.getAllByName(StaticHostProvider.java:92) _[launcher-0.001-SNAPSHOT.jar:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147) [launcher-0.001-SNAPSHOT.jar:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375) [launcher-0.001-SNAPSHOT.jar:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn_SendThread.run(ClientCnxn.java:1137) [launcher-0.001-SNAPSHOT.jar:_]
2023-09-14T13:43:48.628Z WARN apache.zookeeper.ClientCnxn_SendThread ReadOnlyZKClient-vrni-platform-release:2181@0x5c0d876c-SendThread(vrni-platform-release:2181) run:1246 Session 0x0 for server vrni-platform-release:2181, unexpected error, closing socket connection and attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address vrni-platform-release:2181 because it's not resolvable
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71) _[launcher-0.001-SNAPSHOT.jar:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39) _[launcher-0.001-SNAPSHOT.jar:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn_SendThread.startConnect(ClientCnxn.java:1087) _[launcher-0.001-SNAPSHOT.jar:_]
        at org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn_SendThread.run(ClientCnxn.java:1139) [launcher-0.001-SNAPSHOT.jar:_]
2023-09-14T13:43:48.728Z WARN hbase.zookeeper.ReadOnlyZKClient_ZKTask_1 ReadOnlyZKClient-vrni-platform-release:2181@0x5c0d876c exec:192 0x5c0d876c to vrni-platform-release:2181 failed for list of /hbase, code = CONNECTIONLOSS, retries = 11
2023-09-14T13:43:49.728Z ERROR zookeeper.client.StaticHostProvider ReadOnlyZKClient-vrni-platform-release:2181@0x5c0d876c-SendThread(vrni-platform-release:2181) resolve:154 Unable to resolve address: vrni-platform-release:2181
java.net.UnknownHostException: vrni-platform-release


4.  Navigate to live log location /home/ubuntu/build-target/launcher

Here you will see 3 files such as below:

hbase_metric_store.configuration
tsdb_client.configuration
elasticsearch.configuration

Content of above file shows older name "vrni-platform-release"
ubuntu@aria-networks-platform:~/build-target/launcher$ cat tsdb_client.configuration
tsdbHosts: "vrni-platform-release"
tsdbPort: 55111
tsdbShards: 60
ubuntu@aria-networks-platform:~
ubuntu@aria-networks-platform:~/build-target/launcher$ cat elasticsearch.configuration
host_port {
  host: "localhost"
  port: 9300
}
cluster_name: "elasticsearch"
dedup_max_hits_fetch_mk: 6000000
dedup_max_hits_no_mk: 10000000
indexer_threads: 4
zookeeper_connection_string: "vrni-platform-release:2181"
ubuntu@aria-networks-platform:~

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Aria Operations for Networks 6.10.0
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.x
VMware Aria Operations for Networks 6.11.0
VMware vRealize Network Insight 6.9.0
VMware vRealize Network Insight 6.x

Cause

During upgrade launcher service got upgraded first and reconfigured. Then starts infra-base upgrade where we update default hostname for aria rebranding. Launcher is left with old config as hostname updated later. 

Resolution

This issue will be fixed in VMware Aria Operations for Networks version 6.12

Workaround:

To workaround this issue, follow steps mentioned below:

1. SSH to Aria Operation for Network (vRNI) platform appliance as support.

2. Execute below 3 commands:
ub
cd /home/ubuntu/build-target/server-config-mgmt

/bin/bash /home/ubuntu/build-target/server-config-mgmt/run-server-config-mgmt.sh /home/ubuntu/build-target/launcher/server-config.configuration.single /home/ubuntu/build-target/launcher

sudo service launcher-service restart

3. Validate launcher-service has restarted by executing below command/script.
  ./check-service-health.sh -p -d

4. On Aria Operation for Network (vRNI) GUI, all errors should now be cleared for Launcher Service Not Running and no activity errors. 
Also alerts email notification should now not trigger for Launcher Service Not Running and no activity.

Additional Information

Impact/Risks:
VMware Aria Operations for Networks version 6.11.0