The sosrest service on SDDC Manager reports the error "Convert from db to json failed"
search cancel

The sosrest service on SDDC Manager reports the error "Convert from db to json failed"

book

Article ID: 366949

calendar_today

Updated On: 04-25-2025

Products

VMware Cloud Foundation 4.x VMware Cloud Foundation 5.x

Issue/Introduction

  • There is a high number of running tasks in the SDDC UI with no associated task descriptions.
  • SDDC backup task may be hung at 'Backup SDDC Manager System Configuration' subtask.
  • The sosrest service start/restart operation fails.
  • The sosrest service is running but reporting the error "Convert from db to json failed". This can cause the vcf-sos.log and /var/log/messages files to grow and fill up the space on SDDC's root disk partition (/).


A similar error is observed when checking the status of the sosrest service:

systemctl status sosrest.service -l

mm dd Hr:min:sec sosrest[1861]: During handling of the above exception, another exception occurred:
mm dd Hr:min:sec  sosrest[1861]: Traceback (most recent call last):
mm dd Hr:min:sec  sosrest[1861]:   File "framework/workflowhandler.py", line 154, in get_workflow_status
mm dd Hr:min:sec  sosrest[1861]:   File "framework/dbinterface/db_api.py", line 415, in db_to_json
mm dd Hr:min:sec  sosrest[1861]: framework.dbinterface.db_api.DBException: Converting from db to json failed
mm dd Hr:min:sec  sosrest[1861]: During handling of the above exception, another exception occurred:
mm dd Hr:min:sec  sosrest[1861]: Traceback (most recent call last):
mm dd Hr:min:sec  sosrest[1861]:   File "sosapi/controllers/statusapi2.py", line 72, in sos_status
mm dd Hr:min:sec  sosrest[1861]:   File "framework/workflowhandler.py", line 166, in get_workflow_status
mm dd Hr:min:sec  sosrest[1861]: AssertionError: Convert from db to json failed

A similar error is observed in the vcf-sos.log:

/var/log/vmware/vcf/sddc-support/vcf-sos.log:

mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::return_a_session::70::MainThread] DB commit failed
mm dd Hr:min:sec INFO [vcf_sos] [db_api.py::return_a_session::73::MainThread] Closing session
mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::db_to_json::413::MainThread] Converting to JSON from db failed
mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::db_to_json::414::MainThread] Traceback (most recent call last):
  File "/opt/vmware/sddc-support/services/../framework/../dependency/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/vmware/sddc-support/services/../framework/../dependency/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: database disk image is malformed

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

One of the required files for the sosrest service is out of sync 

Potential causes:

  • A disk partition is filling up on the SDDC appliance or a storage event.
  • The sos service was stopped/restarted during an sos operation.

Resolution

Note: This will remove the backup history from SDDC Manager

This requires changes in the SDDC manager database. Kindly ensure to take a snapshot of the SDDC Manager before making the changes

  • SSH into the SDDC Manager Appliance with the vcf user and then elevate to root with su
  • Stop the sosrest service using systemctl stop sosrest.service
  • Move the corrupted files out of the sddc-support directory using below commands

          mv /opt/vmware/vcf/sddc-support/soservice.db /home/vcf

     mv /opt/vmware/vcf/sddc-support/status.json /home/vcf

     mv /opt/vmware/vcf/sddc-support/.status-tmp.json /home/vcf

  • Remove the sos service tasks from Task Aggregator using the below commands 

    /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_url like '%sos%'"

    /usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_type='SDDCMANAGER_BACKUP'"

 

  • Alternative step to remove Task Aggregator 
  • Navigate to postgres database: psql -h localhost -U postgres
  • Navigate to Platform using \c
  • remove Task Aggregator entries 

    delete from task_metadata where task_url like '%sos%';

    delete from task_metadata where task_type='SDDCMANAGER_BACKUP';

 

  • Start the sosrest service using systemctl start sosrest.service
  • Cycle the SDDC services using /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
  • Verify the UI is working as expected and the "Fetching Task" entries have resolved

 

Powered by Wolken Software