Too many open files caused SpectroSERVER crash


Article ID: 208248


Updated On:


CA Spectrum


From /var/log/messages:

 SpectroSERVER[36747]: segfault at e8 ip 00007fa35f2297c5 sp 00007fa2a31d5c20 error 4 in[7fa35f15e000+152000]


Jan 04 12:00:08 ERROR TRACE at table open failed  @
      (0x18)  Db::open: Too many open files
Jan 04 12:00:08 ERROR TRACE at Online Backup database re-opening has failed!. Please shutdown
and re-start the SpectroSERVER.

from stack dump:

Program terminated with signal 11, Segmentation fault.
#0  0x00007fa35f2297c5 in Db::get(DbTxn*, Dbt*, Dbt*, unsigned int) () from /opt/SPECTRUM/lib/
Missing separate debuginfos, use: debuginfo-install glibc-2.17-307.el7.1.x86_64 libgcc-4.8.5-39.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64 nss-softokn-freebl-3.44.0-8.el7_7.x86_64
[?1034h(gdb) where
#0  0x00007fa35f2297c5 in Db::get(DbTxn*, Dbt*, Dbt*, unsigned int) () from /opt/SPECTRUM/lib/
#1  0x00007fa35f223d22 in Dal::TableImpl::select(Dal::Record&) const () from /opt/SPECTRUM/lib/
#2  0x00007fa35f21e9ee in Dal::Table::select(Dal::Record&) const () from /opt/SPECTRUM/lib/
#3  0x00007fa35f1e4e2e in CsDbLandscapeRep::read_mattr_val(CsModelHandle const&, unsigned int, CsAttrDesc::CsAttrType_e, void**) () from /opt/SPECTRUM/lib/
#4  0x00007fa35f1edb81 in CsDbMAttrRep::get_value(CsAttrDesc::CsAttrType_e) () from /opt/SPECTRUM/lib/
#5  0x00007fa36479d577 in CsModel::get_mem_attr_val(unsigned int) const () from /opt/SPECTRUM/lib/../SS/
#6  0x00007fa36479d608 in CsModel::dup_mem_attr_val(CsAttrDesc const*) const () from /opt/SPECTRUM/lib/../SS/
#7  0x00007fa36479ed3b in CsModel::get_int_attr_val(CsMTypeAttr const*) const () from /opt/SPECTRUM/lib/../SS/
#8  0x00007fa3647c0208 in CsModelIf::read(unsigned int) const () from /opt/SPECTRUM/lib/../SS/
#9  0x00007fa36367c108 in CsVnmModelHandle::read(unsigned int) const () from /opt/SPECTRUM/lib/../SS/
#10 0x00007fa36472a021 in CsIHOnlineBackup::ContinueVNM(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/
#11 0x00007fa364730eb9 in CsIHOnlineBackup::backup_database_copy(CsModelHandle const&) () from /opt/SPECTRUM/lib/../SS/
#12 0x00007fa364732fab in CsIHOnlineBackup::performOnlineBackup(CsIHOnlineBackup*) () from /opt/SPECTRUM/lib/../SS/
#13 0x00007fa364419211 in moot_thread_start () from /opt/SPECTRUM/lib/
#14 0x00007fa358488190 in ?? () from /lib64/
#15 0x0000000000000000 in ?? ()


External connection requests on 14023 port causes the Socket FDs growth.

SpectroSERVER process is configured with 1024 Max Open files

If the FD's open files ((ls -l /proc/<ss_pid>/fd | wc -l) exceeds the 1024, the SS process might crash.



Release : 20.2.5

Component : Spectrum Core / SpectroSERVER


Spectrum patch