VCSA VAMI Backup fails with error "Failed to create backup directory"

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

VAMI Backup reports "Failed to create backup directory on backup server."
In /var/log/vmware/applmgmt/backup.log you see the following:

YYYY-MM-DDTHH:MM:SS [20240109-234916-22617221] [MainProcess:PID-902069] [SftpStorageIOLib::mkdir:SftpStorageIOLib.py:352] ERROR: Parent dir sftp://BACKUPFQDN/folder/folder create failed. RC: 21 Err: curl: (21) mkdir command failed: Operation failed

YYYY-MM-DDTHH:MM:SS [20240109-234916-22617221] [MainProcess:PID-902069] [BackupManager::CreateTargetBackupDir:BackupManager.py:112] ERROR: Failed to create backup directory Parent dir sftp://BACKUPFQDN/folder/folder create failed

From the /var/log/vmware/applmgmt/backup.log we observe the following log snippets:

[SftpStorageIOLib::_authorizeLocalHost:SftpStorageIOLib.py:174] ERROR: Failed to authorize local host with command: sshpass -f /tmp/tmp5sdtf_1m ssh-copy-id -i /root/.ssh/id_rsa.pub -p 22 -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts xxxxxxxx@xxxxxxx. Will proceed with backup through curl
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts 'xxxx'@'xxxxxxx' echo ~
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [SftpStorageIOLib::mkdir:SftpStorageIOLib.py:352] ERROR: Parent dir sftp://xxxxxxxx/vCenter create failed. RC: 21 Err: curl: (21) mkdir command failed: Permission denied
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [BackupManager::CreateTargetBackupDir:BackupManager.py:112] ERROR: Failed to create backup directory Parent dir sftp://xxxxxx/vCenter create failed
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [BackupManager::main:BackupManager.py:591] ERROR: BackupManager encountered an exception: Failed to create backup directory on backup server.
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [BackupManager::main:BackupManager.py:596] INFO: skip HandleBackupCleanup as backup target directory creation failed
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
YYYY-MM-DDTHH:MM:SS [20240726-094947-23929136] [MainProcess:PID-43017] [BackupManager::main:BackupManager.py:618] INFO: Backup job failed.

Environment

VMware vCenter Server 8.0.x

VMware vCenter Server 7.0.3

Cause

A change in the FTP backup script causes the backup to fail if the parent directory isn't accessible for the backup path. Even though sub folders will be reachable by the backup user.
Not all the required permissions are provided to the configured SFTP server.

Resolution

VMware is aware of this issue and working towards a fix in a future release.

Workaround:

To work around this issue, modify the /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py file to ignore access denied on the root directory.

Create a backup of the file /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py

cp /usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/plugins/FtpStorageIOLib.py /root/FtpStorageIOLib.py

Edit the file with vi and update the following line at the bottom of the script inside the dir_exists method and change the highlighted line.

    def dir_exists(self, relative_loc='', timeout=DEFAULT_CALL_TIMEOUT_SEC):
        url = self._get_dir_url(self._get_full_url(relative_loc))
        ls_cmd = self._curl_obj.GetCmd(['--list-only', url])
        cmd_res = _run_cmd(ls_cmd, timeout=timeout)
        if cmd_res.returncode == 0:
            return True
        # PR 3025413: Including curl(79) SSH_LAYER_ERROR to check if dir exists.
        if (cmd_res.returncode in [
                CurlEC.DOWNLOAD_FAILED, CurlEC.REMOTE_FILE_NOT_FOUND,
                CurlEC.REMOTE_ACCESS_DENIED, CurlEC.SSH_LAYER_ERROR]):
            return False
        self._process_curl_output(cmd_res)

CHANGE

if cmd_res.returncode == 0:

TO

if (cmd_res.returncode in [0, CurlEC.REMOTE_ACCESS_DENIED]):

Save the file by pressing "esc" and typing :wq!

Ensure the following required permissions are provided to the SFTP server: