Creating a scheduled VCSA backup fails via VAMI with error "Error in method invocation name 'ErrorCodes' is not defined"

Products

• VMware vCenter Server

Issue/Introduction

Attempting to create a scheduled vCenter file backup via the VAMI fails with the following error: 

• "Error in method invocation name 'ErrorCodes' is not defined"

The following can be seen in the vCenter's applmgmt log located /var/log/vmware/applmgmt/applmgmt.log:

• YYYY-MM-DDTHH:MM:SS [9340]DEBUG:vmware.appliance.backup_restore.schedule_impl:Validating schedule spec.
  YYYY-MM-DDTHH:MM:SS [9340]ERROR:plugins.MountedFSStorageIOLib:Cmd ['/usr/bin/mount', '-t', 'nfs', '<BackupServerFQDN>:/<vCenterFQDN>/', '<BackupFolderLocation>', '-o', 'hard,intr'] timed out with timeout 120 seconds. Err: Command '['/usr/bin/mount', '-t', 'nfs', '<BackupServerFQDN>:/<vCenterFQDN>', '<BackupFolderLocation>', '-o', 'hard,intr']' timed out after 120 seconds
  YYYY-MM-DDTHH:MM:SS [9340]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.appliance.recovery.backup.schedules in create - name 'ErrorCodes' is not defined
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py", line 117, in _run_timed_cmd
      out, err = proc.communicate(input=input, timeout=timeout)
    File "/usr/lib/python3.10/subprocess.py", line 1154, in communicate
      stdout, stderr = self._communicate(input, endtime, timeout)
    File "/usr/lib/python3.10/subprocess.py", line 2022, in _communicate
      self._check_timeout(endtime, orig_timeout, stdout, stderr)
    File "/usr/lib/python3.10/subprocess.py", line 1198, in _check_timeout
      raise TimeoutExpired(
  subprocess.TimeoutExpired: Command '['/usr/bin/mount', '-t', 'nfs', '<BackupServerFQDN>:/<vCenterFQDN>/', '<backupFolderLocation>', '-o', 'hard,intr']' timed out after 120 seconds
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py", line 202, in mount
      rc, out, err = _run_timed_cmd(mount_cmd, timeout=MOUNT_TIMEOUT)
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/MountedFSStorageIOLib.py", line 126, in _run_timed_cmd
      raise PluginError(ErrCodes.timeout)
  plugins.StorageIOErrorsLib.PluginError: Plugin error occurred. ErrCode: 7, Args: ()
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
      method_result = self._invoke_int(
    File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 249, in _invoke_int
      method_result = iface.invoke(ctx, method_id, input_value)
    File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
      meth_output = method(**meth_args)
    File "/usr/lib/applmgmt/base/py/vmware/vherd/base/thread_utils.py", line 43, in wrapper
      return sync_call_from_thread(func, *args, **kw)
    File "/usr/lib/applmgmt/base/py/vmware/vherd/base/thread_utils.py", line 13, in sync_call_from_thread
      return threads.blockingCallFromThread(reactor, func, *args, **kw)
    File "/usr/lib/python3.10/site-packages/twisted/internet/threads.py", line 120, in blockingCallFromThread
      result.raiseException()
    File "/usr/lib/python3.10/site-packages/twisted/python/failure.py", line 504, in raiseException
      raise self.value.with_traceback(self.tb)
  NameError: name 'ErrorCodes' is not defined

Environment

VMware vCenter Server 8.x

Cause

VAMI file backup is failing due to the networking connection to the remote backup server timing out.

Resolution

This issue is due to an upstream networking problem causing the VAMI connection to the remote log server to time out. Consult with your networking team to confirm that communication is open between the vCenter and the remote server via ports 111 and 2049. Reference Common ports for protocols used in VAMI based backups.