After upgrade to 20.2, SDC is taking a lot of CPU

Products

• CA Spectrum

Issue/Introduction

We have multiple SDC configured (at least 20) on one server. They are set up as per the documentation on using multiple SDC. 

We notice that once-per-day , some of the SDC are consuming high CPU and is affecting the server performance. What is causing this?

Environment

Release : 20.2

Component : Spectrum Core / SpectroSERVER

Cause

This is a unique scenario which causes this high CPU issue for SDCs. In this specific environment, there are multiple SDCs configured to connect back to a single node SDM on the SpectroSERVER. In addition, there are regular daily VMmotion snapshots occurring on each server. 

After some time troubleshooting, it was determined that the VMmotion snapshot would pause the server, and in doing so the SDC logging would get interrupted and the time stamps would become out-of-sync. Any open sockets that existed prior to the VM Snapshot and subsequent server pause could not be closed, due to the timeshift and incorrect timestamps - the SDC is unable to close the opened sockets, and the sockets left open start consuming CPU. 

Resolution

The issue is resolved temporarily by restarting the SDC service. This forces all sockets to close. However, the issue will return on the next VM Snapshot. 

Another temporary resolution is to halt the VM Snapshots.

A fix has been provided by Broadcom in a patch - Spectrum_10.04.02.02.D140

If you believe you are experiencing this issue, please contact Broadcom Support.