Changing the NFS archive storage location sends alerts for old NFS mount in Aria Operations for Logs

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Receiving alerts about archive failure to the old location. The new location is accepting the data correctly.
• An alert with the following details is present:

Action is required. VMware Aria Operations for Logs failed to connect to the archive storage: nfs://abc.xyz/loginsight-archive. Data might not be able to archive. This could indicate that the archiving destination is unavailable or having performance issues. If this problem is not resolved, the disk on your VMware Aria Operations for Logs installation will fill up and stop accepting data. Other unexpected issues may also occur. Please check the status of the archive destination.

• The /var/log/vmware/loginsight/runtime.log shows entries similar to:

[2023-06-14 15:44:33.845+0000] ["HealthCollectorScheduler-thread-1"/172.16.xx.xxx ERROR] [com.vmware.loginsight.commons.FileUtils] [Failed to run 'mount abc.xyz:/loginsight-archive /storage/core/loginsight/nfsmount/archive-monitor22387283844579839 ' exitCode=32 stdout='' stderr='mount.nfs: No route to host]

 

Environment

VMware Aria Operations for Logs 8.12.x and later.

Cause

VMware Aria Operations for Logs node is still trying to mount an old archive and failing, resulting in error notification.

Resolution

To resolve the issue, choose one of the options below to remove the old archive location from the VMware Aria Operations for Logs internal configuration.

• /internal/config UI
• Command Line

/internal/config UI

1. Navigate to the https://primary_ip_fqdn/internal/config UI in a web browser.

Note: Replace primary_ip_fqdn with the IP address or FQDN of the Primary node.
Example: https://192.168.3.10/internal/config

2. Log in as the local admin user.
3. In the Server Settings Configuration window, find and delete the repository configuration that points to the old archive location.

Example: 
<repository>
<archive-directory value="nfs://abc.xyz/loginsight-archive" />
</repository>

4. Click Save at the bottom of the page.
5. Navigate to the Management > Cluster page.
6. Click Restart next to a worker node and wait for the node status to go back to Connected.
7. Repeat step 6 on all other worker nodes, then finally on the Primary node.

 

Command Line

1. Log into the Primary node as root via SSH or Console.
2. Open storage/core/loginsight/config/loginsight-config.xml#number using a text editor.

Note: Replace number with the largest number file in that directory.  You can use the ls command to see all of the files in the directory.

3. In the file, find and remove the section that is still referencing the old NFS path.

Example:
<repository>
<archive-directory value="nfs://abc.xyz/loginsight-archive" />
</repository>

4. Save and close the file.
5. Run the following command to reboot the node:

reboot -f

7. Once it is rebooted, verify that the old NSF path is no longer located in the storage/core/loginsight/config/loginsight-config.xml#number file.
8. Repeat steps 1-7 on all other nodes in the VMware Aria Operations for Logs cluster.

Additional Information

Impact/Risks:
It is recommended to take a snapshot of the VMware Aria Operations for Logs cluster before proceeding with the Resolution section.