NFS archiving failures troubleshooting guide for vRealize Log Insight

Products

• VMware Aria Suite

Issue/Introduction

This article provides steps for troubleshooting NFS archiving failures in vRealize Log Insight.

Environment

VMware vRealize Log Insight 8.x

Resolution

Definitions

• nfs_path - the NFS path which is used by vRealize Log Insight for archival.
• nfs_host - the host FQDN which exports nfs_path.
• mount_point - the temporary mount point where the NFS share is mounted in vRealize Log Insight.

Troubleshooting

Following commands should be executed on vRealize Log Insight nodes which fail to mount NFS shares.

1. Check if NFS host is reachable:

ping nfs_host

Note: If it fails, there is a networking problem.

2. Check if the desired path is exported on the host:

showmount -e nfs_host

Note: If nfs_path is not returned by this command, it means nfs_path is wrong.

3. If NFSv3 is used on the host, see if RPCs are enabled:

rpcinfo nfs_host | egrep "service|nfs"

Note: If it fails NFS service might be misconfigured.

After checking connectivity and configuration, proceed to mounting the NFS share. 

1. Create a temporary folder where NFS share should be mounted:

mkdir mount_point

2. After executing any of the mount commands below in the Mount Commands section without error, check if the NFS share has been mounted by listing the directory:

ls mount_point

3. Verify content of the NFS share, or check if its available in the list of mounts:

mount -l | grep nfs_host

4. Do not forget to unmount NFS share after each successful mount command, if another mount command should be used

umount mount_point

5. After troubleshooting is finished, make sure to unmount and delete the temporary folder

• umount mount_point
• rm -r mount_point

Mount Commands

1. Check if mount fails with default settings:

mount -v nfs_path mount_point

Notes:

• -v means verbose, which will give you some information about the point of failure.
• This is the actual command used by the vRealize Log Insight service, and if it succeeds archiving should work fine.

2. Check mounting with explicit NFS version:

• NFSv4

mount -v -t nfs -o "vers=4" nfs_path mount_point

• NFSv3

mount -v -t nfs -o "vers=3" nfs_path mount_point

3. For even more customized NFS options look in the NFS manual pages NFS(5).  All those options are accepted as the -o parameter of the command above.

Example: mount -v -t nfs -o "vers=4,proto=tcp,port=1234" nfs_path mount_point

Note: The above command explicitly states to used NFSv4 with tcp protocol and to use custom 1234 port to establish connection to the server.

4. Eventually NFS mounting should succeed, when the correct options are set.  To make those options persistent, open (or create) the file /etc/nfsmount.conf and add the correct options there.

Example: For the above example it will be:
[ Server “<nfs_host>” ]
vers=4
proto=tcp
port=1234

Note: For more information and options available check manual page NFSMOUNT.CONF(5)

5. Verify that NFS share mounts without specifying extra options (Options set in nfsmount.conf are used):

mount nfs_path mount_point

6. Distribute the nfsmount.conf file to other nodes in the cluster. This step is not required if vRealize Log Insight is in standalone mode.

• Create a file workers and fill it with FQDNs of remaining nodes.
• Execute the rcp command in a loop:

for worker in `cat workers`
do
scp /etc/nfsmount.conf root@$worker:/etc/nfsmount.conf
done

• Remove workers file:

rm workers

You can now test archiving from the vRealize Log Insight UI.