SDDC manual Backup fails with 502 Bad Gateway error
search cancel

SDDC manual Backup fails with 502 Bad Gateway error

book

Article ID: 409485

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • Initiating a manual backup of SDDC manager fails with "An exception 502 Bad Gateway occurred during backup. Please re-try."

  • sosrest.service is in activating state

  • Running command: systemctl status sosrest.service -l gives below output.

* sosrest.service - VMware Cloud Foundation Supportability and Serviceability(SoS) Service

     Loaded: loaded (/etc/systemd/system/sosrest.service; enabled; vendor preset: enabled)

     Active: activating (auto-restart) (Result: exit-code) since YYYY-MM-DDTHH:MM:SS UTC; 15s ago

    Process: 78514 ExecStart=/opt/vmware/sddc-support/sosrest (code=exited, status=1/FAILURE)

   Main PID: #### (code=exited, status=1/FAILURE)

Environment

VCF 5.2.x

Cause

The cause of this issue could be the file /opt/vmware/vcf/sddc-support/status.json is corrupted, deleted, or modified.

/var/log/vmware/vcf/sddc-support/vcf-sos.log

YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [database.py::create_database::131::MainThread] Creating tables ['tasks', 'subtasks', 'br_tasks', 'br_subtasks'] for SOS operations
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [database.py::create_database::134::MainThread] DB init operation completed successfully
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::handle_crash_recovery::255::MainThread] Check and Handle SOS REST application crash
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::get_locks::157::MainThread] Get available lock
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::158::MainThread] trying#60 to get http://localhost/locks after 5 seconds
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::159::MainThread] URL GET: http://localhost/locks
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::167::MainThread] Reason:
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::171::MainThread] Status code: 200
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::get_locks::163::MainThread] Locks : []
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::release_lock::204::MainThread] Release DEPLOYMENT lock
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::get_locks::157::MainThread] Get available lock
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::158::MainThread] trying#60 to get http://localhost/locks after 5 seconds
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::159::MainThread] URL GET: http://localhost/locks
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::167::MainThread] Reason:
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [restutil.py::get::171::MainThread] Status code: 200
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::get_locks::163::MainThread] Locks : []
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [workflowhandler.py::release_lock::221::MainThread] System not locked by backup/restore. No lock has to be released
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [db_api.py::fetch_all_records::237::MainThread] Fetching records for table <class 'framework.dbinterface.database.BRTask'> with status in ['In Progress']
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [db_api.py::return_a_session::65::MainThread] Committing session
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [db_api.py::return_a_session::75::MainThread] Closing session
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [db_api.py::fetch_a_record::264::MainThread] Fetching records for table <class 'framework.dbinterface.database.BRTask'> with id in ['########-bcda-4067-8437-20b81330c0c5']
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [db_api.py::return_a_session::75::MainThread] Closing session
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [db_api.py::fetch_a_record::273::MainThread] No record found for query
YYYY-MM-DDTHH:MM:SS ERROR [vcf_sos] [workflowhandler.py::handle_crash_recovery::282::MainThread] 'status'
YYYY-MM-DDTHH:MM:SS ERROR [vcf_sos] [workflowhandler.py::handle_crash_recovery::283::MainThread] Traceback (most recent call last):
File "framework/workflowhandler.py", line 264, in handle_crash_recovery
KeyError: 'status'
YYYY-MM-DDTHH:MM:SS ERROR [vcf_sos] [sosrest.py::startup::98::MainThread] 'status'
YYYY-MM-DDTHH:MM:SS INFO [vcf_sos] [sosrest.py::startup::99::MainThread] SOS REST service failed while checking for system locks

Resolution

Powered by Wolken Software