Changing/decomissioning the NFS archive storage location sends alerts for old NFS mount in Aria Operations for Logs

Products

VMware Aria Suite

Issue/Introduction

Receiving alerts about archive failure to the old location. The new location is accepting the data correctly.
An alert with the following details is present:

Action is required. VMware Aria Operations for Logs failed to connect to the archive storage: nfs://<NFS-Server-FQDN or IP>/loginsight-archive. Data might not be able to archive. This could indicate that the archiving destination is unavailable or having performance issues. If this problem is not resolved, the disk on your VMware Aria Operations for Logs installation will fill up and stop accepting data. Other unexpected issues may also occur. Please check the status of the archive destination.

The /var/log/vmware/loginsight/runtime.log shows entries similar to:

["HealthCollectorScheduler-thread-1"/###.##.##.# ERROR] [com.vmware.loginsight.commons.FileUtils] [Failed to run 'mount <NFS-Server-FQDN or IP>:/loginsight-archive /storage/core/loginsight/nfsmount/archive-monitor22387283844579839 ' exitCode=32 stdout='' stderr='mount.nfs: No route to host]

Removed/updated the NFS store in Log Management > Index Partitions using the steps in the document Configure an Index Partition however continue to receive an alert for old NFS store even after reboot of the cluster
Decommissioned NFS store and continue to receive alert

VMware Aria Operations for Logs is having trouble archiving data on node <ariaoperationsforlogsfqdn>. Data is stored in a series of buckets, and there are 1 pending buckets on this node to archive when exactly 0 are expected. This could indicate that the archiving destination is unavailable or having performance issues. If this problem is not resolved the disk on your VMware Aria Operations for Logs installation on this node will fill up and stop accepting data.

Environment

VMware Aria Operations for Logs 8.18.x

Cause

VMware Aria Operations for Logs node is still trying to mount an old archive and failing, resulting in error notification.

Resolution

To resolve the issue, choose one of the options below to remove the old archive location from the VMware Aria Operations for Logs internal configuration.

/internal/config UI
Command Line

/internal/config UI

Navigate to the https://primary_ip_fqdn/internal/config UI in a web browser.

Note: Replace primary_ip_fqdn with the IP address or FQDN of the Primary node.

Log in as the local admin user.
In the Server Settings Configuration window, find and delete the repository configuration that points to the old archive location.

Example:
<repository>
<archive-directory value="nfs://<NFS-Server-FQDN or IP>/loginsight-archive" />
</repository>

Click Save at the bottom of the page.
Navigate to the Management > Cluster page.
Click Restart next to a worker node and wait for the node status to go back to Connected.
Repeat step 6 on all other worker nodes, then finally on the Primary node.

Command Line

Log into the Primary node as root via SSH or Console.
Stop loginsight services
```
service loginsight stop
```

Make a copy of the latest version of /storage/core/loginsight/config/loginsight-config.xml#number which is the file with the highest number for example. When making the copy increate the file number by one for example

root@<opsforlogshostname> [ /storage/core/loginsight/config ]# ls -lthr
total 28K
-rw-r--r-- 1 root root   36 Jan 22 14:15 node-token
-rw------- 1 root root 6.5K May 29 10:49 loginsight-config.xml#129
-rw------- 1 root root 7.0K May 29 10:49 loginsight-config.xml#130
-rw------- 1 root root 7.0K May 29 14:02 loginsight-config.xml#131

 cp loginsight-config.xml#131 loginsight-config.xml#132

Using a text editor to modify the newly created file find and remove the section that is still referencing the old NFS path.

Example:
<repository>
<archive-directory value="nfs://<NFS-Server-FQDN or IP>/loginsight-archive" />
</repository>

And also maybe present in the section:

<log-partitioning>
    <partition name="<nameofpartition>" enabled="true" internal="true">
      <filter />
      <retention value="0" />
      <archive enabled="true" value="nfs://<NFS-Server-FQDN or IP" />
    </partition>
    <partition name="" enabled="true">
      <filter />
      <retention value="31536000" />
      <archive enabled="true" value="nfs://<NFS-Server-FQDN or IP" />
    </partition>
  </log-partitioning>

Save and close the file.
Run the following command to reboot the node:

reboot -f

Once it is rebooted, verify that the old NSF path is no longer located in the newly created /storage/core/loginsight/config/loginsight-config.xml#number file.
Repeat steps 1-7 on all other nodes in the VMware Aria Operations for Logs cluster.

Additional Information

Impact/Risks:
It is recommended to take a snapshot of the VMware Aria Operations for Logs cluster before proceeding with the Resolution section.