The sosrest service on SDDC Manager reports the error "Convert from db to json failed" or "Upserting Json to DB failed"
search cancel

The sosrest service on SDDC Manager reports the error "Convert from db to json failed" or "Upserting Json to DB failed"

book

Article ID: 366949

calendar_today

Updated On:

Products

VMware Cloud Foundation 4.x VMware Cloud Foundation 5.x VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

  • There is a high number of running tasks in the SDDC UI with no associated task descriptions.
  • SDDC backup task may be hung at 'Backup SDDC Manager System Configuration' subtask.
  • The health-check operation for the SDDC gets stuck: The subtask status remains "In Progress" for "Get Server Details: Retrieve Hosts' IP Configuration Details" and does not transition to "Successful".
  • The sosrest service start/restart operation fails.
  • The sosrest service is running but reporting the error "Convert from db to json failed" or "Upserting Json to DB failed". This can cause the vcf-sos.log and /var/log/messages files to grow and fill up the space on SDDC's root disk partition (/).
  •  SDDC nfs drive may be reported full


A similar error is observed when checking the status of the sosrest service:

systemctl status sosrest.service -l

mm dd Hr:min:sec sosrest[1861]: During handling of the above exception, another exception occurred:
mm dd Hr:min:sec  sosrest[1861]: Traceback (most recent call last):
mm dd Hr:min:sec  sosrest[1861]:   File "framework/workflowhandler.py", line 154, in get_workflow_status
mm dd Hr:min:sec  sosrest[1861]:   File "framework/dbinterface/db_api.py", line 415, in db_to_json
mm dd Hr:min:sec  sosrest[1861]: framework.dbinterface.db_api.DBException: Converting from db to json failed
mm dd Hr:min:sec  sosrest[1861]: During handling of the above exception, another exception occurred:
mm dd Hr:min:sec  sosrest[1861]: Traceback (most recent call last):
mm dd Hr:min:sec  sosrest[1861]:   File "sosapi/controllers/statusapi2.py", line 72, in sos_status
mm dd Hr:min:sec  sosrest[1861]:   File "framework/workflowhandler.py", line 166, in get_workflow_status
mm dd Hr:min:sec  sosrest[1861]: AssertionError: Convert from db to json failed

Similar errors are observed in the SDDC vcf-sos.log:

/var/log/vmware/vcf/sddc-support/vcf-sos.log:

mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::return_a_session::70::MainThread] DB commit failed
mm dd Hr:min:sec INFO [vcf_sos] [db_api.py::return_a_session::73::MainThread] Closing session
mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::db_to_json::413::MainThread] Converting to JSON from db failed
mm dd Hr:min:sec ERROR [vcf_sos] [db_api.py::db_to_json::414::MainThread] Traceback (most recent call last):
  File "/opt/vmware/sddc-support/services/../framework/../dependency/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/opt/vmware/sddc-support/services/../framework/../dependency/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: database disk image is malformed


/var/log/vmware/vcf/sddc-support/vcf-sos.log

Traceback (most recent call last):
  File "health/general/base.py", line 137, in validate_cert
  File "health/healthutil.py", line 1290, in update_health_progress
  File "utils/progressreporter.py", line 344, in update_task_status
  File "framework/workflowhandler.py", line 658, in push_task_progress_to_db
  File "framework/dbinterface/db_api.py", line 574, in upsert_json
framework.dbinterface.db_api.DBException: Upserting Json to DB failed

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

Potential causes:

  • One of the required files for the sosrest service is out of sync 
  • A disk partition is filling up on the SDDC appliance or a storage event.
  • The sos service was stopped/restarted during an sos operation.

Resolution

Note: This will remove the Backup history and SOS Health Check history from the SDDC Manager.

This requires changes in the SDDC manager database, ensure to take a snapshot of the SDDC Manager before making the changes.

  1. SSH into the SDDC Manager Appliance with the vcf user and then elevate to root with su

  2. Stop the sosrest service using
    systemctl stop sosrest.service
  3. Move the corrupted files out of the sddc-support directory using below commands:
   mv /opt/vmware/vcf/sddc-support/soservice.db /home/vcf
     mv /opt/vmware/vcf/sddc-support/status.json /home/vcf
     mv /opt/vmware/vcf/sddc-support/.status-tmp.json /home/vcf

4. Remove the sos service tasks from Task Aggregator using the below commands:


If on VCF 5.1.1 or higher:

/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_url like '%sos%'"
/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_type='SDDCMANAGER_BACKUP'"

If on VCF 5.1.0 or lower:

psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_url like '%sos%'"
psql -h localhost -U postgres -d platform -c "delete from task_metadata where task_type='SDDCMANAGER_BACKUP'"

5.Start the sosrest service using:

systemctl start sosrest.service

6.Cycle the SDDC services using:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

7.Verify the UI is working as expected and the "Fetching Task" entries have resolved

 

Additional Information

If stale tasks remain after following the above attempt script removal as detailed in folloiwng:

https://knowledge.broadcom.com/external/article/395842/cleanup-stale-tasks-after-sddc-manager-r.html

Powered by Wolken Software