Performing a VCSA backup via VAMI is failing with error "Failed to get etcd snapshot for all supervisor clusters"
search cancel

Performing a VCSA backup via VAMI is failing with error "Failed to get etcd snapshot for all supervisor clusters"

book

Article ID: 377953

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0 VMware vCenter Server 7.0

Issue/Introduction

  • VAMI(vCenter Server Appliance Management) based backup for VCSA (vCenter Server Appliance) fails with the error "Failed to get etcd snapshot for all supervisor clusters"
  • Supervisor Cluster is not configured on the vCenter
  • The issue is same even without the option "Supervisors Control Plane" being checked in the backup data on VAMI
  • /var/log/vmware/applmgmt/backup.log:

YYYY-MM-DDTHH:MM:SS [20240806-110609-23929136] [MainProcess:PID-3623342] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts '<backup user>'@'<backup server IP>' echo ~
YYYY-MM-DDTHH:MM:SS [20240806-110609-23929136] [VTSDBBackup:PID-3623565] [VTSDB::BackupVTSDBCleanup:VTSDB.py:237] INFO: Successfully cleaned up for VTSDB backup.
YYYY-MM-DDTHH:MM:SS [20240806-110609-23929136] [ComponentScriptsBackup:PID-3623562] [Log::run:Log.py:64] ERROR: 404 Client Error: Not Found for url: https://<localhost>:443/wcp
YYYY-MM-DDTHH:MM:SS [20240806-110609-23929136] [ComponentScriptsBackup:PID-3623562] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:87] ERROR: Component backup command "/etc/vmware/backup/component-scripts/wcp/wcp_backup_restore.py --startBackup" failed 1.
YYYY-MM-DDTHH:MM:SS [20240806-110609-23929136] [ComponentScriptsBackup:PID-3623562] [Log::run:Log.py:64] ERROR: Failed to get etcd snapshot for all supervisor clusters: Failed to generate SAML token and create sessionFailed to complete backup: Failed to generate SAML token and create session
YYYY-MM-DDTHH:MM:SS [20240806-110609-23929136] [ComponentScriptsBackup:PID-3623562] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:115] ERROR: Error during component wcp backup
Underlying process status. rc: 1
stdout:
stderr:
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ComponentScripts.py", line 91, in ComponentScriptsBackup
    raise BackupRestoreError(('Error during component %s backup' %
util.Common.BackupRestoreError: Error during component wcp backup
Underlying process status. rc: 1
stdout:

  • /var/log/vmware/wcp/wcpsvc-backup-restore.log:

YYYY-MM-DDTHH:MM:SS [3874414][MainThread]ERROR:wcp_backup_restore:The operation failed with error Failed to generate SAML token and create session
Traceback (most recent call last):
  File "/etc/vmware/backup/component-scripts/wcp/authentication.py", line 124, in get_session
    self._session = session_svc.create()
  File "/usr/lib/applmgmt/vapi/lib/vapi_common_client-2.100.0.egg/com/vmware/cis_client.py", line 202, in create
    return self._invoke('create', None)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 266, in native_invoke
    method_result = self.invoke(ctx, method_id, data_val)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 199, in invoke
    return self._api_provider.invoke(self._iface_id.get_name(),
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/security/client/security_context_filter.py", line 101, in invoke
    method_result = ApiProviderFilter.invoke(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/filter.py", line 75, in invoke
    method_result = self.next_provider.invoke(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/protocol/client/msg/json_connector.py", line 79, in invoke
    response = self._do_request(VAPI_INVOKE, ctx, params)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/protocol/client/msg/json_connector.py", line 158, in _do_request
    http_response.data.raise_for_status()
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://localhost:443/wcp

Resolution

If the issue is encountered, please help to collect a log bundle for vCenter server and create a Broadcom Support case to identify the cause of the issue.

As a workaround, perform the below steps

  • Log in to the VCSA via ssh session
  • Navigate to the below location

cd /etc/vmware/backup/component-scripts/wcp/

  • Take a backup of the wcp_backup_restore.py script in the same location

cp wcp_backup_restore.py wcp_backup_restore.py.old

  • Manually edit the file using vi editor, we are making the script to bypass the wcp component in the script for wcp. 

vi wcp_backup_restore.py

Existing Line

 try:
     logger.info("Backup script called with opts: %s" % args)
     set_up_logging(os.path.join(LOG_PATH, "wcpsvc-backup-restore.log"))
     if args.op == "startBackup":
         backup()
     elif args.op == "stopBackup":

Updated Line

 try:
     logger.info("Backup script called with opts: %s" % args)
     set_up_logging(os.path.join(LOG_PATH, "wcpsvc-backup-restore.log"))
     if args.op == "startBackup":
         #backup()
         pass
     elif args.op == "stopBackup":

Note: These steps will be modifying the python script and hence the spacing as mentioned above is critical

  • Save and exit the editor using wq!
  • restart the applmgmt service using the below command

service-control --restart applmgmt

  • Proceed with the backup via VAMI
Powered by Wolken Software