File based Backup configuration of vCenter server via VAMI using NFS protocol : Error in method invocation 'PluginError' object has no attribute 'errorcode'

Products

• VMware vCenter Server

Issue/Introduction

• Unable to configure VAMI Backup, it will error out with: Error in method invocation 'PluginError' object has no attribute 'errorcode'

• /var/log/vmware/applmgmt/backup.log shows below events
  
  2023-11-06T19:57:47 PM AEDT [2466]ERROR:plugins.MountedFSStorageIOLib:Cmd ['/usr/bin/mount', '-t', 'nfs', 'cohesity.xx.xxxx.xx:/Vcenter_Backup/', '/storage/remote/backup/nfs/cohesity.xx.xxxx.xx/xIu3pnLJ/6Qf7BLX7', '-o', 'hard,intr'] timed out with timeout 120 seconds. Err: Command '['/usr/bin/mount', '-t', 'nfs', 'cohesity.xx.xxxx.xx:/Vcenter_Backup/', '/storage/remote/backup/nfs/cohesity.xx.xxxx.xx/xIu3pnLJ/6Qf7BLX7', '-o', 'hard,intr']' timed out after 120 seconds

2023-11-06T19:57:47 PM AEDT [2466]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.appliance.recovery.backup.schedules in create - 'PluginError' object has no attribute 'errorcode' subprocess.TimeoutExpired: Command '['/usr/bin/mount', '-t', 'nfs', 'cohesity.xx.xxxx.xx:/Vcenter_Backup/', '/storage/remote/backup/nfs/cohesity.xx.xxxx.xx/xIu3pnLJ/6Qf7BLX7', '-o', 'hard,intr']' timed out after 120 seconds.

Environment

VMware vCenter Server 7.x

Cause

This looks like an issue with the network connectivity between the server hosting the NFS share and the vCenter Server.

Resolution

There may be firewall issues blocking the connectivity between NFS and vCenter. Please check with the customer if all the ports are open.

The issue could be resolved by addressing the firewalls that are blocking the connectivity between the NFS server and vCenter Server.

Since NFS mounts generally work and only fail during the creation of VAMI backup configurations, it is unlikely that firewall issues are the cause. Instead, it appears to be an issue on the NFS server end. To further validate this theory and confirm if there are no responses from the server, please capture packets of the failed NFS mount, which will help verify any potential firewall issues.

Alternatively, we can increase the MOUNT_Timeout parameter in the MountedFSStorageIOLib.py script.

Please follow the steps below:

1. Take a snapshot without memory for the vCenter Server VM if standalone or offline snapshot of all vCenters if they are in ELM.
2. SSH to the vCenter Server as root, enter shell.
3. Take a backup of this file: cp /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py /tmp/MountedFSStorageIOLib.py.back
4. Increase the MOUNT_TIMEOUT and set it to 200 seconds.

Please run the commands below:

• vi /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py
• Hit I for insert.
• Find the entry : MOUNT_TIMEOUT and change the value from 120 to 200.
• Exit the file by : click Esc then type :wq!

5. Restart vCenter services: service-control --stop --all && service-control --start --all
6. Initiate the backup again and provide an update.

Additional Information

Impact/Risks:
File Based Backup will be impacted.