Performing a VCSA backup via VAMI is failing with error "Failed to get etcd snapshot for all supervisor clusters"

Products

• VMware vCenter Server

Issue/Introduction

• VAMI(vCenter Server Appliance Management) based backup for VCSA (vCenter Server Appliance) fails with the error "Failed to get etcd snapshot for all supervisor clusters"
• Supervisor Cluster is not configured on the vCenter
• The issue is same even without the option "Supervisors Control Plane" being checked in the backup data on VAMI
• Log location on vCenter Server - /var/log/vmware/applmgmt/backup.log:

YYYY-MM-DDTHH:MM:SS [YYYYMMDD-######-#######] [MainProcess:PID-3623342] [SftpStorageIOLib::_readRemoteRoot:SftpStorageIOLib.py:147] INFO: Executing command to get remote root: ssh -q -p 22 -i /root/.ssh/id_rsa -o UserKnownHostsFile=/root/.ssh/br_temp_known_hosts '<backup user>'@'<backup server IP>' echo ~
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-######-#######] [VTSDBBackup:PID-3623565] [VTSDB::BackupVTSDBCleanup:VTSDB.py:237] INFO: Successfully cleaned up for VTSDB backup.
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-######-#######] [ComponentScriptsBackup:PID-3623562] [Log::run:Log.py:64] ERROR: 404 Client Error: Not Found for url: https://<localhost>:443/wcp
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-######-#######] [ComponentScriptsBackup:PID-3623562] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:87] ERROR: Component backup command "/etc/vmware/backup/component-scripts/wcp/wcp_backup_restore.py --startBackup" failed 1.
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-######-#######] [ComponentScriptsBackup:PID-3623562] [Log::run:Log.py:64] ERROR: Failed to get etcd snapshot for all supervisor clusters: Failed to generate SAML token and create sessionFailed to complete backup: Failed to generate SAML token and create session
YYYY-MM-DDTHH:MM:SS [YYYYMMDD-######-#######] [ComponentScriptsBackup:PID-3623562] [ComponentScripts::ComponentScriptsBackup:ComponentScripts.py:115] ERROR: Error during component wcp backup
Underlying process status. rc: 1
stdout:
stderr:
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/components/ComponentScripts.py", line 91, in ComponentScriptsBackup
    raise BackupRestoreError(('Error during component %s backup' %
util.Common.BackupRestoreError: Error during component wcp backup
Underlying process status. rc: 1
stdout:

• /var/log/vmware/wcp/wcpsvc-backup-restore.log:

YYYY-MM-DDTHH:MM:SS [3874414][MainThread]ERROR:wcp_backup_restore:The operation failed with error Failed to generate SAML token and create session
Traceback (most recent call last):
  File "/etc/vmware/backup/component-scripts/wcp/authentication.py", line 124, in get_session
    self._session = session_svc.create()
  File "/usr/lib/applmgmt/vapi/lib/vapi_common_client-2.100.0.egg/com/vmware/cis_client.py", line 202, in create
    return self._invoke('create', None)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 266, in native_invoke
    method_result = self.invoke(ctx, method_id, data_val)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 199, in invoke
    return self._api_provider.invoke(self._iface_id.get_name(),
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/security/client/security_context_filter.py", line 101, in invoke
    method_result = ApiProviderFilter.invoke(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/provider/filter.py", line 75, in invoke
    method_result = self.next_provider.invoke(
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/protocol/client/msg/json_connector.py", line 79, in invoke
    response = self._do_request(VAPI_INVOKE, ctx, params)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/protocol/client/msg/json_connector.py", line 158, in _do_request
    http_response.data.raise_for_status()
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://localhost:443/wcp

Environment

VMware vCenter Server 7.x 

VMware vCenter Server 8.x 

VMware vCenter Server 9.x 

Resolution

If the issue is encountered, collect a log bundle for vCenter Server and create a Broadcom Support case to identify the cause of the issue.

As a workaround, perform the below steps

1. Log in to the VCSA via ssh session
2. Navigate to the below location
   cd /etc/vmware/backup/component-scripts/wcp/


3. Take a backup of the wcp_backup_restore.py script in the same location
   cp wcp_backup_restore.py wcp_backup_restore.py.old


4. Manually edit the file using vi editor, we are making the script to bypass the wcp component in the script for wcp.
   
   

   Existing Line

    try:
     logger.info("Backup script called with opts: %s" % args)
     set_up_logging(os.path.join(LOG_PATH, "wcpsvc-backup-restore.log"))
     if args.op == "startBackup":
         backup()
     elif args.op == "stopBackup":

   

   Updated Line

    try:
     logger.info("Backup script called with opts: %s" % args)
     set_up_logging(os.path.join(LOG_PATH, "wcpsvc-backup-restore.log"))
     if args.op == "startBackup":
         #backup()
         pass
     elif args.op == "stopBackup":

   

   Note: These steps will be modifying the python script and hence the spacing as mentioned above is critical

5. Save and exit the editor using wq! 
6. Restart the applmgmt service using the below command
   service-control --restart applmgmt
   
   
7. Proceed with the backup via VAMI