vCenter Server 8.0 Update 3i/j File-Based Backups fail with "server validation failure" (RC 21)
search cancel

vCenter Server 8.0 Update 3i/j File-Based Backups fail with "server validation failure" (RC 21)

book

Article ID: 442896

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • After upgrading to vCenter Server 8.0 Update 3i (8.0.3 P08) or Update 3j (8.0.3 P09), both manual and scheduled file-based backups are failing
  • The VAMI UI displays an error message:
    Backup job failed to start due to server validation failure.
  • The /var/log/vmware/applmgmt/backup.log remains empty or shows no new entries during the failed attempt
  • During a failed scheduled backup, the following error stack can be found in /var/log/vmware/applmgmt/backupScheduler.log:
    <timestamp> [0] [MainProcess:PID-###] [Scheduler::ExecScheduleRun:Scheduler.py:137] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.appliance.unknown-twisted-error', default_message='Internal error. See logs for details. https://<VC_FQDN>/appliance/support-bundle', args=['<VC_FQDN>'], params=None, localized=None)], data : None, error_type : None}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1197, in run
    return self._invoke('run',
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.appliance.unknown-twisted-error', default_message='Internal error. See logs for details. https://<VC_FQDN>/appliance/support-bundle', args=['<VC_FQDN>'], params=None, localized=None)], data : None, error_type : None}
  • In both scenarios, manual and scheduled backup, /var/log/vmware/applmgmt/applmgmt.log has the following entries:
    <timestamp> [9687]ERROR:plugins.FtpStorageIOLib:sftp cmd failed. RC: 21, Err: curl: (21) mkdir command failed: Permission denied, Cmd: ['/usr/bin/curl', '--fail', '-u', '<backup_user>:****', '--connect-timeout', '120', '--ssl-reqd', '-k', '--noproxy', '*', '--silent', '--show-error', '-Q', 'mkdir /<target_folder>/backup_<timestamp>/' 'sftp://<backup_server_FQDN>/<backup_folder>/'], stdout:
<timestamp> [9687]ERROR:backupRestoreAPI:Backup Server validation failed with PluginError: PluginError('Plugin error occurred. ErrCode: 102, Args: ()')
..
<timestamp> [9687]ERROR:backupRestoreAPI:Backup Server validation failed for scheduled backup: Plugin error occurred. ErrCode: 102, Args: ()
<timestamp> [9687]ERROR:backupRestoreAPI:Validation failure error message: Structure('LocalizableMessage', dict(id='com.vmware.applmgmt.backup.failed_start_validation', defaultMessage='Backup job failed to start due to server validation failure.', args=[]))
<timestamp> [9687]ERROR:backupRestoreAPI:set message: [defaultdict(<function ConfigManager.defaultFactory at 0x7fd58d38d7e0>, {'id': 'com.vmware.applmgmt.backup.failed_start_validation', 'defaultMessage': 'Backup job failed to start due to server validation failure.', 'args': []})]

Environment

VMware vCenter Server 8.0 Update 3i and later versions

Cause

vCenter 8.0 Update 3i introduced a mandatory pre-backup validation script. This script attempts to create a dummy folder on the SFTP target named backup_YYYYMMDD-HHMM to verify write permissions before data collection begins. If this fails, the actual backup process will not be initiated, therefore no entries can be seen in backup.log, as the pre-validation script logs its actions directly to the appliance management service log (/var/log/vmware/applmgmt/applmgmt.log)

If the SFTP user account has insufficient permissions, specifically missing the Create Directory (mkdir) right, the SFTP server will retunr Return Code (RC) 21 (Permission denied). The validation script treats this as a critical failure and aborts the backup to prevent data loss or incomplete transfers.

Resolution

To solve this issue, verify that the user configured in the backup does have sufficient access to the backup server.

The permissions required for the user are:

  • Create Directory
  • Write/Upload
  • Read/Download

If these permission are correctly set, confirm if security policies, user quotas, or "Chroot" environment settings on the SFTP server were modified recently.

To verify the correct configuration, run a manual directory creation test from the vCenter command line using the backup credentials:

# curl -u <user>:<password> sftp://<sftp-ip>/<path>/ -Q "mkdir /<path>/test-folder"

If this returns curl: (21) mkdir command failed: Permission denied, the issue must be resolved on the SFTP server side.

Additional Information

If multiple vCenters use the same SFTP target and start backups at the same minute, a collision may occur because the dummy folder name is not unique per vCenter. In that scenario, stagger backup schedules by 5 to 10 minutes. For more information, refer to Scheduled file-based backups periodically fail with "server validation failure" when multiple vCenter Server 8.0 Update 3i or later versions are using the same backup target.

See also File-Based Backup and Restore of vCenter Server for more information about the file-based backup feature.

Powered by Wolken Software