vCenter server's file level backup fails with 'ERROR: sftp quote cmd failed. Err: curl: (9) Could not open directory for reading: Permission denied' when using 'completeFTP' as backup solution.

Products

• VMware vCenter Server

Issue/Introduction

• Unable to perform vCenter server's file level backup to the sftp file location configured using completeFTP as backup solution. 
• Saving the backup configuration will succeed however backup task will be failed immediate after initiation. 
• The backup failure indicating that the configured SFTP user or the directory itself  does not have sufficient directory creation or write privileges on the SFTP server path however all permissions set for user account and directory. 
  
  /var/log/vmware/applmgmt/backup.log 
  

  YYYY-MM-DDTHH:MM:SS.###  [ConfigFilesBackup:PID-835514] [Proc::GetProcsStatus:Proc.py:352] ERROR: rc: 1, stderr: Traceback (most recent call last):
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 58, in <module>
      main(sys.argv[1], sys.argv[2] sys.argv[3]
    File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/../util/Calculate.py", line 45, in main
      stdout_obj.write(data)

  BrokenPipeError: [Errno 32] Broken pipe

  YYYY-MM-DDTHH:MM:SS.###  [MainProcess:PID-835281] [BackupManager::Cleanup:BackupManager.py:452] INFO: Cleaning up failed backup files.
  YYYY-MM-DDTHH:MM:SS.### [MainProcess:PID-835281] [FtpStorageIOLib::_process_curl_output:FtpStorageIOLib.py:143] ERROR: sftp quote cmd failed. Err: curl: (9) Could not open directory for reading: Permission denied,
  Cmd: ['/usr/bin/curl', '--fail', '-u', '<Domain>\\<sftp_user>:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--proxy', 'http://localhost:1083/', '--proxytunnel', '--silent', '--show-error', 'sftp://<sftp_backup server>/directory_path']
  YYYY-MM-DDTHH:MM:SS.###  [MainProcess:PID-835281] [BackupManager::Cleanup:BackupManager.py:455] WARNING: Ignore to clean up failed backup files, due to issue: 'Plugin error occurred. ErrCode: 3, Args: ()'.
  YYYY-MM-DDTHH:MM:SS.### [MainProcess:PID-835281] [BackupManager::HandleBackupCleanup:BackupManager.py:499] ERROR: Failed to cleanup the backup job. Error: Failed to clean up the backup job.
  YYYY-MM-DDTHH:MM:SS.### [ [MainProcess:PID-835281] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
  YYYY-MM-DDTHH:MM:SS.### [MainProcess:PID-835281] [BackupManager::main:BackupManager.py:678] INFO: Backup job failed.
  
  

Environment

VCF 9.x 

vCenter server 8.0.x 

Cause

• Depending on the operation, backup destination configured with 'CompleteFTP' returns two different statuses for the non-existent path causing the backup operation to fail. 

WARN SFTPSubsystemServer Sending SSH_FXP_STATUS [requestid=1,code=10,msg=invalid path]

WARN SFTPSubsystemServer  Sending SSH_FXP_STATUS [requestid=1,code=3,msg=permission denied]

Resolution

• To resolve the issue occurs due to a limitation of concurrent SSH Sessions on the SFTP Server, follow the steps: 'Backup Manager encountered an exception' error during vCenter VAMI Backup using SFTP
  
  
• If the backup server has security configurations to restrict the number of consecutive login failures, the VC connection might get locked and backup server will not accept any further requests. 
  To resolve this restriction in sftp server, follow this steps to skip the ssh tunnel creation for SFTP backups : VAMI SFTP backup fails while creating backup directory on remote backup server
  
  
• If issue persists, follow below given steps. 
  
  >>Download the attached patch file 'dir_exists.patch'.
  
  >>Take a backup of FtpStorageIOLib.py:
       cp /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py.bak

  >>Execute the patch file within the root directory:
       cd / && patch -p1 < dir_exists>.patch

  >>Restart applmgmt service:
       service-control --restart applmgmt

  >>Retry performing the backup task.