SpectroSERVER crashing due to conflicting HashNodes while loading the Evpn and Vpn modules.

book

Article ID: 7406

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction

The SpectroSERVER crashed while loading the Enterprise VPN Module.

We are seeing a lot of the following in the VNM.OUT:

 mts free: heap -1 unallocated address 0x7f70c1d02c5d on page 0x7f70245c8000 (size >= 60K objects)

The stack from the generate core files contained:

/lib64/libc.so.6[0x3778e75dee]
/lib64/libc.so.6[0x3778e78c3d]
/opt/SPECTRUM/lib/../SS/libmdlsvint.so.1(_ZN18GlobalCollectionIH16reconcile_searchERK13CsModelHandleP15CsModelDescList+0x578)

[0x7fd3595a64e8]
/opt/SPECTRUM/lib/../SS/libmdlsvint.so.1(_ZN18GlobalCollectionIH10run_searchERK13CsModelHandleNS_17SearchInitiator_eE+0x138)

[0x7fd3595a74b8]
/opt/SPECTRUM/lib/../SS/libsskrnl.so.1(_ZN12CsIHTimerMgr16do_periodic_workEPK18CsPeriodicWorkNode+0x574)[0x7fd35b449cc4]
/opt/SPECTRUM/lib/libwkmgr.so.1(_ZN23CsPeriodicWorkScheduler17process_work_nodeEP10CsWorkNode+0x81)[0x7fd3568f4651]
/opt/SPECTRUM/lib/libwkmgr.so.1(_ZN15CsWorkScheduler7do_workEv+0xda)[0x7fd3568f53aa]
/opt/SPECTRUM/lib/libmoot.so.1(moot_thread_start+0x23)[0x7fd35b134843]
/lib64/libc.so.6[0x3778e43760]

Cause

The instability was caused by duplicate definitions of HashNode in two different generics - Evpn, and Vpn. These two generics don't reference or link with each other, so one might think this isn't an issue. 

However, both these modules get loaded into the same process, and these are 'weak' symbols, and quietly collide. Sustaining Engineering's guess is that generally, by luck, the larger object (one is 12 bytes, while the other one is 8 bytes) wins, and this also was really a bug exposed by the x64 project as now pointers are bigger.

Environment

Release: SDBSFO99000-10.2-Spectrum-Device Based Suite-Server FOC
Component:

Resolution

Spectrum_10.2.0.D230 changes the names of HashNode in each generic, to be unique, and also takes an additional step of changing how we load Modules so that this isn't a problem in the future. 

 

 

This will be included in Spectrum 10.2.2 and Spectrum 10.3.