Analytics Service Fails to Start On Multi Node Cluster in vRealize Operations Manager 6.x and later
search cancel

Analytics Service Fails to Start On Multi Node Cluster in vRealize Operations Manager 6.x and later

book

Article ID: 343471

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:

When there is more than 1 node in the cluster and the status of 1 or more nodes is Waiting for Analytics in the admin UI, you experience these symptoms:

  • In the /storage/vcops/log/analytics-UUID.log file, you see entries similar to:

    2016-10-10 15:17:03,539 INFO [Analytics Main Thread ] com.vmware.statsplatform.persistence.sqldb.SQLDBHAManagerImpl.waitForOtherNodesToPublishMasterDB - 4 out of total 5 have published its master db knowledge
     
  • In the /storage/vcops/log/cassandra/system.log file, you see entries similar to:

    ERROR [WrapperSimpleAppMain] 2016-10-10 15:18:52,294 DatabaseDescriptor.java:141 - Fatal configuration error
    org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:120) ~[apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:136) ~[apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:164) [apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.8.jar:2.1.8] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_102]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_102]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_102]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_102]
    at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290) [wrapper-3.5.24.jar:3.5.24]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
    Caused by: org.yaml.snakeyaml.reader.ReaderException: special characters are not allowed
    at org.yaml.snakeyaml.reader.StreamReader.checkPrintable(StreamReader.java:93) ~[snakeyaml-1.11.jar:na]
    at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:192) ~[snakeyaml-1.11.jar:na]
    at org.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:60) ~[snakeyaml-1.11.jar:na]
    at org.yaml.snakeyaml.Yaml.load(Yaml.java:412) ~[snakeyaml-1.11.jar:na]
    at org.apache.cassandra.config.YamlConfigurationLoader.logConfig(YamlConfigurationLoader.java:126) ~[apache-cassandra-2.1.8.jar:2.1.8]
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:104) ~[apache-cassandra-2.1.8.jar:2.1.8]
    ... 12 common frames omitted


    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.


Environment

VMware vRealize Operations Manager 6.2.x
VMware vRealize Operations Manager 6.4.x
VMware vRealize Operations Manager 7.5.x
VMware vRealize Operations Manager 6.3.x
VMware vRealize Operations Manager 6.7.x
VMware vRealize Operations Manager 6.5.x
VMware vRealize Operations Manager 6.1.x
VMware vRealize Operations Manager 7.0.x
VMware vRealize Operations Manager 6.6.x

Cause

This issue occurs because the $VCOPS_BASE/user/conf/cassandra/cassandra.yaml file has incorrect information.

Resolution

To resolve the issue, copy the $VCOPS_BASE/user/conf/cassandra/cassandra.yaml file, from a working node to the problematic node and update the information.

  1. Log in to the problematic vRealize Operations Manager node as root through SSH or Console.
  2. Run this command to rename the cassandra.yaml file:

    mv /usr/lib/vmware-vcops/user/conf/cassandra/cassandra.yaml /usr/lib/vmware-vcops/user/conf/cassandra/cassandra.yaml.bak
     
  3. Copy the following files from a working node, to the $VCOPS_BASE/user/conf/cassandra/ directory on the problematic node.

    $VCOPS_BASE/user/conf/cassandra/cassandra.yaml
    $VCOPS_BASE/user/conf/cassandra/cqlshrc


    Note: The scp command, or an SCP utility can be used for the RHEL and vApp versions of vRealize Operations Manager.
     
  4. Open the $VCOPS_BASE/user/conf/cassandra/cassandra.yaml file, using a text editor.
  5. Replace the IP Address of the node from where you copied cassandra.yaml file with the IP address of node where you copied to.

    Note: The sed command can be used for the RHEL and vApp versions of vRealize Operations Manager instead:

    For example:

    sed -i "s/working_node_address/problematic_node_address/g" /usr/lib/vmware-vcops/user/conf/cassandra/cassandra.yaml

    Note: Replace working_node_address with the address of the node the cassandra.yaml file was copied from, and replace problematic_node_address with the address of the problematic node.
     
  6. Log in to vRealize Operations Manager admin UI.
  7. Take the cluster Offline.
  8. Bring the cluster Online.

 


Additional Information

简体中文:在 vRealize Operations Manager 6.x 中,分析服务在多节点群集上启动失败