VIP/ILB showing as disconnected from UI on all nodes except primary in Aria Operations for logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VIP/ILB showing as Unavailable from UI on all nodes except primary.
• System monitor page can't load from any node except primary.
• On the storage/core/loginsight/var/runtime.log for primary node these entries will be found
  [####-##-## 08:10:16.648+0000] ["#######################-######-#"/###.###.###.### WARN] [com.vmware.loginsight.analytics.distributed.SearchResponseAggregator] [Worker failed to return result for query, worker: StrataNodeInfo(host:###.###.###.###, port:16520, token:########-####-####-####-## , status:CONNECTED), query: ]
[####-##-## 08:10:16.648+0000] ["#######################-######-#"/###.###.###.### ERROR] [com.vmware.loginsight.analytics.distributed.AbstractSearchResponseAggregator] [Partial results are being returned as 1 out of 3 nodes failed to respond. Request token . Please contact an admin user for more information.]
[####-##-## 08:10:16.650+0000] ["#######################-######-#"/###.###.###.### INFO] [com.vmware.loginsight.analytics.ErrorUtils] [mark error code=5 (Partial results are being returned as 1 out of 3 nodes failed to respond. Request token . Please contact an admin user for more information.)]
  
  
• On the storage/core/loginsight/var/runtime.log for worker node these entries will be found
  [####-##-## 07:04:10.609+0000] ["#################################-######-#"/###.###.###.### WARN] [com.vmware.loginsight.loadbalancer.LoadBalancerEmbeddedService] [Error trying to read cluster node states [510 suppressed]]
com.vmware.loginsight.cluster.ClusterStateReaderException: Not yet initialized or master unavailable
        at com.vmware.loginsight.cluster.DaemonClusterNodesMaintenanceStatusReader.getClusterNodesMaintenanceStatus(DaemonClusterNodesMaintenanceStatusReader.java:39) ~[#######-###.jar:?]
        at com.vmware.loginsight.loadbalancer.LoadBalancerEmbeddedService.updateLoadBalancerState(LoadBalancerEmbeddedService.java:579) [####-########-#######.jar:?]
        at com.vmware.loginsight.loadbalancer.LoadBalancerEmbeddedService$3.run(LoadBalancerEmbeddedService.java:318) [####-########-#######.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
• Checking the /storage/core/loginsight/config/loginsight-config.xml#[highest_number] file we find that all nodes have the primary host added with FQDN instead of IP address and worker nodes are added with IP addresses.
   <distributed #########-########="true">
    <daemon host="primary_node_fqdn" port="16520" token="########-####-####-####-############">
      <#######-##### name="standalone" />
    </daemon>
    <daemon host="###.###.###.###" port="16520" token="########-####-####-####-############">
      <#######-##### name="workernode" />
    </daemon>
    <daemon host="###.###.###.###" port="16520" token="########-####-####-####-############">
      <#######-##### name="workernode" />
    </daemon>
  </distributed>

Environment

Aria Operations for logs 8.18.x

Cause

The worker nodes are unable to make RPC calls to the primary node due to DNS resolution issues.

Resolution

To mitigate the DNS resolution issues, update the /storage/core/loginsight/config/loginsight-config.xml#[highest_number] file to use the IP address of the primary node instead of its FQDN. Take offline snapshots before proceeding the steps below on ALL nodes:

1. SSH into the node as root user and run the following command to stop the loginsight service on all nodes.

service loginsight stop

2. Locate the .xml config file with the highest number by going to the following directory.

cd /storage/core/loginsight/config

ls -l

Example: Identify the file with the highest number (e.g., loginsight-config.xml#23). Use the highest number specific to your environment as shown in the file list in the screenshot.

3. Run the following command to create a backup of the config file.

cp loginsight-config.xml#23 loginsight-config.xml#23.bkp

4. Run the following command to edit the config file and update the daemon host from priamry node FQDN to its IP address.

vi loginsight-config.xml#[highest_number]

Press i to start inserting and replace:
daemon host="primary_node_FQDN"

With:
daemon host="primary_node_IP_ADDRESS"

5. Press Esc and type the following to save and exit the file.

:wq!

6. Run the following command to start the loginsight service and apply the changes.

service loginsight start

7. Wait for a moment for the VIP/ILB to show connected again.

Note: Ensure that the updated loginsight-config.xml file is correctly replicated on all worker nodes. If the changes are not reflected automatically, please repeat the above steps on each worker node. Once the function is restored, run the following command on ALL nodes to delete loginsight-config.xml#23.bkp created at step 3.

rm loginsight-config.xml#23.bkp