Upgrade from vRLI 8.3 to 8.4 Fails if deployed with FIPS enabled
search cancel

Upgrade from vRLI 8.3 to 8.4 Fails if deployed with FIPS enabled

book

Article ID: 312254

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
  • vRealize Log Insight 8.3 was installed with FIPS mode enabled in the OVF Properties during deployment.
  • Upgrading form vRealize Log Insight 8.3 to 8.4 fails with the error Upgrade Unconfirmed.
  • The /storage/var/loginsight/runtime.log file shows errors similar to:
org.apache.thrift.transport.TTransportException: Failed to obtain the server socket. 
at com.vmware.loginsight.lib.thrift.ThriftServer$SecureTransportCreator.create(ThriftServer.java:196) 
at com.vmware.loginsight.lib.thrift.ThriftServer.makeServerSocket(ThriftServer.java:349) 
at com.vmware.loginsight.lib.thrift.ThriftServer.makeServerSocket(ThriftServer.java:371) 
at com.vmware.loginsight.lib.thrift.ThriftServer.<init>(ThriftServer.java:319) 
at com.vmware.loginsight.lib.thrift.ThriftServer.<init>(ThriftServer.java:403) 
at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startThriftServer$15(LogInsightDaemon.java:640) 
at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:763) 
at com.vmware.loginsight.daemon.LogInsightDaemon.startThriftServer(LogInsightDaemon.java:637) 
at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:330) 
at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1070)
Caused by: org.apache.thrift.transport.TTransportException: Error creating the transport 
at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:228) 
at org.apache.thrift.transport.TSSLTransportFactory.getServerSocket(TSSLTransportFactory.java:114) 
at com.vmware.loginsight.lib.thrift.ThriftServer$SecureTransportCreator.create(ThriftServer.java:186) 
... 9 more 
Caused by: java.io.IOException: Could not load file: /usr/lib/loginsight/application/etc/3rd_config/keystore.bcfks


Environment

VMware vRealize Log Insight 8.3.x
VMware vRealize Log Insight 8.x

Cause

This issue is caused by FIPS mode being enabled in the OVF Properties while deploying a new vRealize Log Insight 8.3 Cluster.
If FIPS mode enabled later from the UI, or vRealize Log Insight was deployed with a release older than 8.3, this article is not applicable.

Resolution

Complete the steps below for your Cluster deployment type.
Quick links:
Single node Cluster
Multi node Cluster
 

Single Node Cluster

  1. Log into the vRealize Log Insight Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
  2. Run the following commands one at a time:
  • cd /usr/lib/loginsight/application/etc/3rd_config 
  • ls | grep keystore.bcfks | sort -n -k1.6 | tail -n1 | xargs -I '{}' mv {} keystore.bcfks 
  • cd /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.63/conf 
  • ls | grep keystore.bcfks | sort -n -k1.6 | tail -n1 | xargs -I '{}' mv {} keystore.bcfks 
  • service loginsight restart
The vRealize Log Insight 8.4 upgrade should now show as Completed.

 

Multi Node Cluster

  1. Log into the vRealize Log Insight Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
  2. Run the following commands one at a time:
  • cd /usr/lib/loginsight/application/etc/3rd_config 
  • ls | grep keystore.bcfks | sort -n -k1.6 | tail -n1 | xargs -I '{}' mv {} keystore.bcfks 
  • cd /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.63/conf 
  • ls | grep keystore.bcfks | sort -n -k1.6 | tail -n1 | xargs -I '{}' mv {} keystore.bcfks 
  • service loginsight restart
Note: The UI of the Primary node should now be up and running.  In the Administration > Cluster page, the Primary node's status will be Connected but, other nodes will be in an Upgrade Pending or Transferring PAK state.
  1. The upgrade needs to be completed manually on the rest of the nodes.  Copy the vRealize Log Insight 8.4 PAK file to the /tmp directory of all Worker nodes using an SCP Utility.
Note: The PAK file is already uploaded to the Primary node under the /tmp directory, so it can easily copied to the Worker nodes using scp command.

Example: From the Primary node, run the following command, replacing worker_ip with the IP address of one of the Worker nodes.  Repeat the command for the other Worker nodes in the cluster:
scp /tmp/VMware-vRealize-Log-Insight-8.4.0-17828109.pak root@worker-ip:/tmp 
  1. Log into the first Worker node as root via SSH or Console, and run this command to start the upgrade:
loginsight-pak-upgrade /tmp/VMware-vRealize-Log-Insight-8.4.0-17828109.pak 
  1. Once the Worker node has restarted from the upgrade, log back in and run the following commands on it, one at a time:
cd /usr/lib/loginsight/application/etc/3rd_config
ls | grep keystore.bcfks | sort -n -k1.6 | tail -n1 | xargs -I '{}' mv {} keystore.bcfks
cd /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.63/conf
ls | grep keystore.bcfks | sort -n -k1.6 | tail -n1 | xargs -I '{}' mv {} keystore.bcfks
service loginsight restart
  1. Repeat steps 4-5 on all other Worker nodes in the cluster.
  2. All nodes should now be up and running, however the Administration > Cluster page, may still show their states as Upgrade Pending or Transferring PAK.
  3. Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
  4. Run the following command to get the Cassandra credentials:
/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/credentials-look-up
  1. You will receive output similar to the following.  Note the password value in-between the quotation marks.
<cassandra-user value="lisuper" />
<cassandra-password value="nwbnurx5_S" />
  1. Run the following command to update the Cluster status:
/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/cqlsh -u lisuper -p "cassandra_password" --cqlshrc=/storage/core/loginsight/cidata/cassandra/config/cqlshrc -e "update logdb.upgrade_status set status='Complete' where version = '8.4.0-17828109';"

Note: Replace cassandra_password with the password noted in step 10.
  1. Run the following command to obtain the node ID for each node in the cluster:
/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/cqlsh -u lisuper -p "cassandra_password" --cqlshrc=/storage/core/loginsight/cidata/cassandra/config/cqlshrc -e "select node_id from logdb.node_upgrade_status;"

Note: Replace cassandra_password with the password noted in step 10.
  1. You will see output similar to the following.  Node the ID of each node:
node_id 
-------------------------------------- 
<UUID_1> 
<UUID_2>
<UUID_3>
  1. Run the following command to set the node's upgrade status to Complete:
/usr/lib/loginsight/application/lib/apache-cassandra-*/bin/cqlsh -u lisuper -p "cassandra_password" --cqlshrc=/storage/core/loginsight/cidata/cassandra/config/cqlshrc -e "update node_upgrade_status set status='Complete' where version = '8.4.0-17828109' and node_id in ('id-1', 'id-2', 'id-3')"

Note: Replace cassandra_password with the password noted in step 10, and replace id-1, id-2, id-3 with the node IDs noted in step 13.  There may be more or fewer IDs depending on you cluster size.  List all IDs separated by a comma and surrounded by single quotes.

Example: /usr/lib/loginsight/application/lib/apache-cassandra-*/bin/cqlsh -u lisuper -p "nwbnurx5_S" --cqlshrc=/storage/core/loginsight/cidata/cassandra/config/cqlshrc -e "update node_upgrade_status set status='Complete' where version = '8.4.0-17828109' and node_id in ('<UUID_1>', '<UUID_2>', '<UUID_3>')"

The vRealize Log Insight 8.4 upgrade should now show as Completed.


Powered by Wolken Software