vPostgres PANIC : could not fsync file "pg_tblspc/XXX/XXX/XXX/XXX": Input/output error

Products

• VMware vCenter Server

Issue/Introduction

• Unable to access vCenter
• "Failed to connect to database" from /var/log/vmware/vpxd/vpxd.log
  
  • _{YYYY-MM-DDTHH:MM:SS error vpxd[3808370] [Originator@6876 sub=vpxdVdb] [VpxdVdb::SetDBType] Failed to connect to database: ODBC error: (08001) - [unixODBC]connection to server on socket "/var/run/vpostgres/.s.PGSQL.xxxx" failed: No such file or directory}
• PANIC reported from /var/log/vmware/vpostgres/postgresl.log
  
  • _{YYYY-MM-DD HH:MM:SS UTC 67d3d22c.387aad 38146294 VCDB vpxd [local] 3701421 6PANIC:  could not fsync file "pg_tblspc/XXX/XXX/XXX/XXX": Input/output error
    YYYY-MM-DD HH:MM:SS UTC 67d3d22c.387aad 38146294 VCDB vpxd [local] 3701421 7STATEMENT:   truncate table vpx_hist_stat1_82 
    YYYY-MM-DD HH:MM:SS UTC 67d3ed53.397ca4 38147132 VCDB vpxd [local] 3767460 4LOG:  duration: 408950.452 ms  plan:
    ... ...
    YYYY-MM-DD HH:MM:SS UTC 67d3c337.37d3bb 38146748 VCDB vpxd [local] 3658683 17ERROR:  could not extend file "pg_tblspc/XXX/XXX/XXX/XXX": Read-only file system
    YYYY-MM-DD HH:MM:SS UTC 67d3c337.37d3bb 38146748 VCDB vpxd [local] 3658683 18HINT:  Check free disk space.}
• "Buffer I/O error on device" from journalctl
  • _{./commands/journalctl_-b--1.txt:MM DD HH:MM:SS vcenter kernel: Buffer I/O error on device dm-5, logical block 3711335
    ./commands/journalctl_-b--1.txt:MM DD HH:MM:SS vcenter kernel: Buffer I/O error on device dm-13, logical block 2503856}

 

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue could be caused by:

• Disk space is full
• Storage performance

 

Resolution

Investigate storage performance referring https://knowledge.broadcom.com/external/article/344099/using-esxtop-to-identify-storage-perform.html