VCSA backup via VAMI post upgrading to 8.0 U3 using sftp might fail with the error "Error in method invocation"
search cancel

VCSA backup via VAMI post upgrading to 8.0 U3 using sftp might fail with the error "Error in method invocation"

book

Article ID: 377806

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

  • vCenter Server Appliance(VCSA) backup via vCenter Server Appliance Management Interface (VAMI) fails with the error "Error in method invocation"

  • Issue is seen if ssh connection from vCenter server to backup server is not allowed
  • /var/log/vmware/applmgmt/applmgmt.log:

YYYY-MM-DDTHH:MM:SS PM UTC [2505280]DEBUG:vmware.vherd.base.proxy_utils:Proxy is not enabled.
YYYY-MM-DDTHH:MM:SS PM UTC [2505280]ERROR:vmware.vapi.provider.local:Error in invoking com.vmware.appliance.recovery.backup.schedules in update - Command '['ssh-keyscan', '-p', '<sftp port>', '-H', '<backup server FQDN/IP>']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 277, in invoke
    method_result = self._invoke_int(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/local.py", line 249, in _invoke_int
    method_result = iface.invoke(ctx, method_id, input_value)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/skeleton.py", line 371, in invoke
    meth_output = method(**meth_args)
  File "/usr/lib/applmgmt/base/py/vmware/vherd/base/thread_utils.py", line 43, in wrapper
    return sync_call_from_thread(func, *args, **kw)
  File "/usr/lib/applmgmt/base/py/vmware/vherd/base/thread_utils.py", line 13, in sync_call_from_thread
    return threads.blockingCallFromThread(reactor, func, *args, **kw)
  File "/usr/lib/python3.10/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "/usr/lib/python3.10/site-packages/twisted/python/failure.py", line 488, in raiseException
    raise self.value.with_traceback(self.tb)
subprocess.CalledProcessError: Command '['ssh-keyscan', '-p', '<sftp port>', '-H', '<backup server FQDN/IP>']' returned non-zero exit status 1.

  • /var/log/vmware/applmgmt/applmgmt.log:

YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-2387589] [Scheduler::ExecScheduleRun:Scheduler.py:137] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.appliance.unknown-twisted-error', default_message='Internal error. See logs for details. https://<VCSA FQDN>/appliance/support-bundle', args=['<VCSA FQDN>'], params=None, localized=None)], data : None, error_type : None}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1197, in run
    return self._invoke('run',
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.appliance.unknown-twisted-error', default_message='Internal error. See logs for details. https://<VCSA FQDN>/appliance/support-bundle', args=['<VCSA FQDN>'], params=None, localized=None)], data: None, error_type : None}
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-2474079] [Scheduler::ExecArchiveList:Scheduler.py:119] ERROR: Failed to issue the Archive.list request. Exception: {messages : [LocalizableMessage(id='vapi.method.invoke.exception', default_message="Error in method invocation Command '['ssh-keyscan', '-p', '<sftp port>', '-H', '<backup server FQDN/IP>']' returned non-zero exit status 1.", args=["Command '['ssh-keyscan', '-p', '<sftp port>', '-H', '<backup server FQDN/IP>']' returned non-zero exit status 1."], params=None, localized=None)], data : None, error_type : INTERNAL_SERVER_ERROR}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 116, in ExecArchiveList
    backupList = svc_handle.list(locationSpec, systemName, filterSpec)
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup/system_name_client.py", line 289, in list
    return self._invoke('list',
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.InternalServerError: {messages : [LocalizableMessage(id='vapi.method.invoke.exception', default_message="Error in method invocation Command '['ssh-keyscan', '-p', '<sftp port>', '-H', '<backup server FQDN/IP>']' returned non-zero exit status 1.", args=["Command '['ssh-keyscan', '-p', '<sftp port>', '-H', '<backup server FQDN/IP>']' returned non-zero exit status 1."], params=None, localized=None)], data : None, error_type : INTERNAL_SERVER_ERROR}

Cause

The issue is caused due to the ssh communication from the vCenter to the backup server is not available causing the "ssh-keyscan" operation to fail

Resolution

This issue is fixed in vCenter Server 8.0 Update 3e. Please see 8.0 U3e (24674346) release notes (PR 3439353)

  • Description: SFTP backups might fail to establish a connection with the backup server. In the applmgmt.log/backup.log, you see an error such as:"Command '['ssh-keyscan', '-p', 'port'', '-H', 'hostname']' returned non-zero exit status 1."

===============================================================

Workaround

To workaround the issue without patching to build with fix, implement the steps from either of the below options.

Option 1

Proceed with enabling ssh communication to the backup server from vCenter server on the port configured for backup

Option 2

  • Log in to the vCenter server via ssh
  • Navigate to the below location

cd /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/

  • Take a backup of the file

cp SftpStorageIOLib.py SftpStorageIOLib.py.copy

  • Manually edit the file using vi editor

vi SftpStorageIOLib.py

Existing Line

if self.validateCerts:
    self.knownHostFile = BR_CERT_KNOWN_HOSTS_FILE
else:
    self.knownHostFile = KNOWN_HOSTS_FILE
self._setupPasswordlessTunnel(user, passwd)

Updated Line

if self.validateCerts:
    self.knownHostFile = BR_CERT_KNOWN_HOSTS_FILE
else:
    self.knownHostFile = KNOWN_HOSTS_FILE
try:
    self._setupPasswordlessTunnel(user, passwd)
except Exception as e:
    self.transferWithCurl = True

Note: These steps will be modifying the python script and hence the spacing as mentioned above is critical

  • Save and exit the editor using wq!
  • restart the applmgmt service using the below command

service-control --restart applmgmt

  • Proceed with the backup via VAMI

 

Additional Information

Powered by Wolken Software