Too many open files caused SpectroSERVER crash
Article ID: 208248
Updated On:
Products
Issue/Introduction
From /var/log/messages:
SpectroSERVER[36747]: segfault at e8 ip 00007fa35f2297c5 sp 00007fa2a31d5c20 error 4 in libssdbl.so.1[7fa35f15e000+152000]
From VNM.OUT:
Jan 04 12:00:08 ERROR TRACE at CsSSDbRp.cc(658): table open failed @ TableImpl.cc:661
(0x18) Db::open: Too many open files
Jan 04 12:00:08 ERROR TRACE at CsIHOB.cc(1812): Online Backup database re-opening has failed!. Please shutdown
and re-start the SpectroSERVER.
from stack dump:
Program terminated with signal 11, Segmentation fault.
#0 0x00007fa35f2297c5 in Db::get(DbTxn*, Dbt*, Dbt*, unsigned int) () from /opt/SPECTRUM/lib/libssdbl.so.1
Missing separate debuginfos, use: debuginfo-install glibc-2.17-307.el7.1.x86_64 libgcc-4.8.5-39.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 nss-softokn-freebl-3.44.0-8.el7_7.x86_64
[?1034h(gdb) where
#0 0x00007fa35f2297c5 in Db::get(DbTxn*, Dbt*, Dbt*, unsigned int) () from /opt/SPECTRUM/lib/libssdbl.so.1
#1 0x00007fa35f223d22 in Dal::TableImpl::select(Dal::Record&) const () from /opt/SPECTRUM/lib/libssdbl.so.1
#2 0x00007fa35f21e9ee in Dal::Table::select(Dal::Record&) const () from /opt/SPECTRUM/lib/libssdbl.so.1
#3 0x00007fa35f1e4e2e in CsDbLandscapeRep::read_mattr_val(CsModelHandle const&, unsigned int, CsAttrDesc::CsAttrType_e, void**) () from /opt/SPECTRUM/lib/libssdbl.so.1
#4 0x00007fa35f1edb81 in CsDbMAttrRep::get_value(CsAttrDesc::CsAttrType_e) () from /opt/SPECTRUM/lib/libssdbl.so.1
#5 0x00007fa36479d577 in CsModel::get_mem_attr_val(unsigned int) const () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#6 0x00007fa36479d608 in CsModel::dup_mem_attr_val(CsAttrDesc const*) const () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#7 0x00007fa36479ed3b in CsModel::get_int_attr_val(CsMTypeAttr const*) const () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#8 0x00007fa3647c0208 in CsModelIf::read(unsigned int) const () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#9 0x00007fa36367c108 in CsVnmModelHandle::read(unsigned int) const () from /opt/SPECTRUM/lib/../SS/libIHapi.so.1
#10 0x00007fa36472a021 in CsIHOnlineBackup::ContinueVNM(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#11 0x00007fa364730eb9 in CsIHOnlineBackup::backup_database_copy(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#12 0x00007fa364732fab in CsIHOnlineBackup::performOnlineBackup(CsIHOnlineBackup*) () from /opt/SPECTRUM/lib/../SS/libsskrnl.so.1
#13 0x00007fa364419211 in moot_thread_start () from /opt/SPECTRUM/lib/libmoot.so.1
#14 0x00007fa358488190 in ?? () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
Cause
External connection requests on 14023 port causes the Socket FDs growth.
SpectroSERVER process is configured with 1024 Max Open files
If the FD's open files ((ls -l /proc/<ss_pid>/fd | wc -l) exceeds the 1024, the SS process might crash.
Environment
Release : 20.2.5
Component : Spectrum Core / SpectroSERVER
Resolution
Spectrum patch 10.04.02.02.D104
Attachments
Feedback
