spectrum ds synch failing

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration

Issue/Introduction

the spectrum data source is not synching, it goes from synching to failed. no new events are being seen in spectrum from CAPC. 

In the logs we see that the PULL phase of the data source synch kicks off with a session id of x
then, a few seconds later another PULL kicks off with a session id of y.

​log sample:
Data source Spectrum Infrastructure [email protected] encountered an error while processing a sync request.  The problem is in the data source, not CAPC.  Check the logs for the data source to determine the cause of the problem.  The following stack trace shows the context of the sync request:
javax.xml.ws.soap.SOAPFaultException: PullRequest: invalid client session identifier: deafd3c2-27e2-4621-99af-037a9f05c063
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:156)
    
...............
Data source Spectrum Infrastructure [email protected] encountered an error while processing a sync request.  The problem is in the data source, not CAPC.  Check the logs for the data source to determine the cause of the problem.  The following stack trace shows the context of the sync request:
javax.xml.ws.soap.SOAPFaultException: PullRequest: invalid client session identifier: 886225f8-4d88-4feb-85e5-e7a687dd6239
    at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:156)
    at com.sun.proxy.$Proxy103.pullRequest(Unknown Source)

Cause

There was a duplicate CAPM environment which is supposed to be inactive, used for disaster recovery.
During an upgrade, services were accidentally left running on the duplicate PC.
So Device Manager service 1 would start a pull and issue session ID x.
Device Manager service 2 would start a pull with session ID y.
When session x replied, the session ID was invalid so communication failed.

Environment

CAPC 3.x

Resolution

Stopped and disabled the secondary environment.