The sosrest service on SDDC Manager reports error, "Convert from db to json failed"

Products

• VMware Cloud Foundation 4.x
• VMware Cloud Foundation 5.x

Issue/Introduction

• There is a high number of Running tasks in the SDDC UI with no associated task descriptions.
• SDDC backup task may be hung at 'Backup SDDC Manager System Configuration' subtask.
• The sosrest service start/restart operation fails.
• The sosrest service is running but reporting error, "Convert from db to json failed". This can lead to the vcf-sos.log and /var/log/messages files to grow and take over the space on SDDC's root disk partition (/).

A similar error is observed when checking the status of the sosrest service. 

systemctl status sosrest.service -l

mm dd Hr:min:sec sosrest[1861]: During handling of the above exception, another exception occurred:
mm dd Hr:min:sec  sosrest[1861]: Traceback (most recent call last):
mm dd Hr:min:sec  sosrest[1861]:   File "framework/workflowhandler.py", line 154, in get_workflow_status
mm dd Hr:min:sec  sosrest[1861]:   File "framework/dbinterface/db_api.py", line 415, in db_to_json
mm dd Hr:min:sec  sosrest[1861]: framework.dbinterface.db_api.DBException: Converting from db to json failed
mm dd Hr:min:sec  sosrest[1861]: During handling of the above exception, another exception occurred:
mm dd Hr:min:sec  sosrest[1861]: Traceback (most recent call last):
mm dd Hr:min:sec  sosrest[1861]:   File "sosapi/controllers/statusapi2.py", line 72, in sos_status
mm dd Hr:min:sec  sosrest[1861]:   File "framework/workflowhandler.py", line 166, in get_workflow_status
mm dd Hr:min:sec  sosrest[1861]: AssertionError: Convert from db to json failed


A similar error is observed in the vcf-sos.log

/var/log/vmware/vcf/sddc-support/vcf-sos.log:

mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::return_a_session::70::MainThread] DB commit failed
mm dd Hr:min:sec INFO [vcf_sos] [db_api.py::return_a_session::73::MainThread] Closing session
mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::db_to_json::413::MainThread] Converting to JSON from db failed
mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::db_to_json::414::MainThread] Traceback (most recent call last):
  File "/opt/vmware/sddc-support/services/../framework/../dependency/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/vmware/sddc-support/services/../framework/../dependency/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: database disk image is malformed

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

One of the required files for the sosrest service is out of sync. 

Potential causes:

• Disk partition filling up on the SDDC appliance or a storage event.
• The sos service was stopped/restarted during a sos operation.

Resolution

This requires changes in SDDC manager database. Please raise a Support Case with Broadcom to receive further assistance.

Note: Snapshot of SDDC Manager is required before implying changes.