VMware vRealize Log Insight Upgrade Prerequisites
search cancel

VMware vRealize Log Insight Upgrade Prerequisites

book

Article ID: 327827

calendar_today

Updated On:

Products

VMware

Issue/Introduction

This article provides upgrade requirements and troubleshooting guidance for a VMware vRealize Log Insight cluster.

If a cluster upgrade is started and becomes stuck part-way through, the cluster may be left in a partially-upgraded state. See Resuming rolling upgrade of a VMware vRealize Log Insight cluster (2123064).

If all requirements described by this article are met but retry of the upgrade fails, collect a log bundle from every node in the cluster and engage VMware Support for further investigation. For more information, see Collecting diagnostic information for VMware vRealize Log Insight (2056760) 

 

Resolution

There are several known scenarios which can cause Log Insight upgrades to fail. Validate whether these scenarios are present in the affected environment and address them, then retry the upgrade.

Log Insight nodes are down or unreachable

Log Insight 2.5 and higher has Apache Cassandra installed on every node. Quorum is required in order to perform many operations, including upgrade. Upgrades are performed one node at a time to maintain quorum. If another node in the cluster is down, the cluster may not be able to attain quorum.

Validate the status of nodes:

  1. Review the status of all Log Insight cluster members in the cluster.

    1. Using a web browser, open the Log Insight administration user interface at https://LogInsightHostnameOrIpAddress/admin/
    2. Click the Cluster tab.
    3. Take note of any node which appears with a status other that Connected.
  2. Review the status of all Cassandra services in the cluster.

    1. Open a console or SSH session to the Log Insight virtual appliance and log in as root.

    2. Switch the current working directory by running this command:

      cd /usr/lib/loginsight/application/lib/apache-cassandra-*/bin

    3. Verify that the cluster status is UP by running this command:

      ./nodetool status

      For example, you may see output similar to:

      Datacenter: datacenter1
      =======================
      Status=Up/Down
      |/ State=Normal/Leaving/Joining/Moving
      -- Address Load Tokens Owns Host ID Rack
      UN 10.11.12.100 151.99 MB 256 19.6% 75fd0bf8bd0556fcc3b073fed6aefb36 rack1
      UN 10.11.12.101 156.46 MB 256 20.6% f79845ca3e7fab13448340b4403ad80f rack1
      UN 10.11.12.102 135.12 MB 256 19.2% fe90f17b7f8ff0f45df7aed699b50fbf rack1
      UN 10.11.12.103 134.84 MB 256 20.6% b1ab7493430c60e9ded8644f66bc4c11 rack1
      DN 10.11.12.104 127.27 MB 256 20.0% 63e4a2884fad9907fe87347c43ba341a rack1

    4. Take note of any node which appears in a status other than UN (Up/Normal). Investigate why that node is not participating in the cluster and take action to restore it:

      • Use the administration user interface cluster page to connect or restart nodes.
      • Restart nodes by rebooting the virtual machine.

NTP is not configured correctly or not synchronized

Log Insight requires that time synchronization is configured correctly on all nodes. If time gets out of sync, this can cause failures of cross-cluster operations. 

DNS resolution of localhost produces a non-loopback address

Validate the content of /etc/hosts on all Log Insight nodes. If localhost resolves to an address other than 127.0.0.1, nodetool may fail to connect to the running Cassandra service.

  1. Open a console or SSH session to the Log Insight virtual appliance and log in as root.
  2. Determine the Log Insight node's IP address using this command:

    /opt/vmware/share/vami/vami_ip_addr

  3. Open the /etc/hosts configuration file in a text editor. For more information, see Editing files on an ESX host using vi or nano (1020302).

  4. Inspect the contents of the /etc/hosts configuration file. A correct configuration lists the Log Insight node's fully-qualified domain name (FQDN) and/or hostname on the same line as the static IP, and localhost on the same line as the loopback address 127.0.0.1. For example:

    10.11.12.13 loginsight-node.example.com loginsight-node
    127.0.0.1 localhost

  5. If the /etc/hosts configuration file contains a line pairing the static IP address with the name localhost, correct it to match the example.

  6. Save and close the /etc/hosts configuration file.

  7. Validate that a DNS lookup for localhost produces the loopback address using the command:

    host localhost

    Expected output:

    localhost has address 127.0.0.1

  8. Restart the Log Insight service using the command:

    service loginsight restart

 

Additional Information

vRealize Log Insight nodes are virtual machines. Prior to an upgrade, we expect that the virtual machines running vRealize Log Insight nodes are not affected by performance bottlenecks or have network connectivity problems when communicating with each other. If you suspect either of these are occurring, review the following KB articles:

Editing files on an ESX host using vi or nano
Collecting diagnostic information for VMware vRealize Log Insight
Resuming rolling upgrade of a VMware vRealize Log Insight cluster


Powered by Wolken Software