SDDC Manager /data partition filling up to 100% frequently

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager /data partition filling up to 100% frequently
• Error in /var/log/vmware/vcf/sddc-support/vcf-sos-svcs.log

  sosrest[1906]: sqlite3.DatabaseError: database disk image is malformed

• Running pg dump steps to know the cause of database error
  • mkdir -p /tmp/backup/postgres
  • cd /tmp/backup/postgres
  • pg_dumpall -h localhost -U postgres -a > postgres.bkp
• pg_dump fails with below errors

  pg_dump: error: Dumping the contents of table "session" failed: PQgetResult() failed.
  pg_dump: error: Error message from server: ERROR: could not open file "base/#####/919045.3" (target block 809476): previous segment is only 18140 blocks
  pg_dump: error: The command was: COPY public.session (sid, sess, expire) TO stdout;
  pg_dumpall: error: pg_dump failed on database "sddc_manager_ui", exiting

• When selecting table session from sddc_manager_ui, it fails to open with the following error

  For VCF 4.x
  psql -h localhost -U postgres -d sddc_manager_ui -c "select * from session"
  
  For VCF 5.x
  /usr/pgsql/13/bin/psql -h localhost -U postgres -d sddc_manager_ui -c "select * from session"

  ERROR: could not open file "base/#####/919045.3" (target block 809476) : previous segment is only 95976 blocks

  Size of session table also appears to be incredibly large in size

Environment

VMware Cloud Foundation

Cause

Corruption in table "session" from "sddc_manager_ui" database

Resolution

The resolution requires clearing out database tables. Please contact Broadcom support  for assistance.