ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Spectrum Data Source sync failure post migration

book

Article ID: 227693

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

The PC Portal and Spectrum OC hosts were migrated to new hosts with new names.

Everything is working using new SSL certificates with the new host names.

The OC SSL root and intermediate certificates were exported from OC and imported to PC Portal.

Trying to Save the Spectrum Data Source in the Portal Web UI after setting it's new host name fails. Sync is also failing.

When trying to Save the Spectrum Data Source in PC Portal it failed with this PCService.log error. It indicates an inability to successfully communicate with the Spectrum OC host. It suggests review of the Spectrum OC host tomcat logs is the next step.

ERROR | qtp1093605336-1770       | 2021-11-02 19:21:08,561 | com.ca.im.portal.site.admin.models.DataSourceAdminModel          
      | Failed to create datasource: Could not send Message.
javax.xml.ws.WebServiceException: Could not send Message.
 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
 at com.sun.proxy.$Proxy114.productID(Unknown Source)
 at com.ca.im.portal.api.services.datasource.ProductSync2WSProxyImpl.getProductId(ProductSync2WSProxyImpl.java:43)
 at com.ca.im.portal.api.services.datasource.DataSourcePoll.getDataSourceGuid(DataSourcePoll.java:419)
 at com.ca.im.portal.api.services.datasource.DataSourcePoll.pollProductSyncWS(DataSourcePoll.java:140)
 at com.ca.im.portal.api.services.datasource.DataSourcePoll.poll(DataSourcePoll.java:98)
 at com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl.buildImpl(DataSourceBuilderImpl.java:149)
 at com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl.build(DataSourceBuilderImpl.java:125)
 at com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl.commitWithId(DataSourceBuilderImpl.java:187)
 at com.ca.im.portal.site.admin.models.DataSourceAdminModel.create(DataSourceAdminModel.java:655)
 at com.ca.im.portal.site.admin.models.DataSourceAdminModel.handleFormSubmit(DataSourceAdminModel.java:934)
 at com.ca.im.portal.render.desktop.report.FormRenderer.processFormSubmit(FormRenderer.java:131)
 at com.ca.im.portal.render.desktop.report.FormRenderer.requestFromClient(FormRenderer.java:99)
 at com.ca.im.portal.render.desktop.BaseRenderer.handleRequestFromClient(BaseRenderer.java:398)
 at com.ca.im.portal.render.desktop.DesktopRenderMgr.processClientRequest(DesktopRenderMgr.java:323)
 at com.ca.im.portal.desktop.PageData.doPost(PageData.java:190)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
 at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
 at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
 at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
 at com.ca.im.portal.common.web.security.RequestFilter.doFilter(RequestFilter.java:27)
 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
 at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
 at com.ca.im.portal.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.server.Server.handle(Server.java:516)
 at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
 at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
 at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
 at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
 at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking https://<Spectrum_OC_Host>:8443/axis2/services/ProductSync2WS: Read timed out
 at sun.reflect.GeneratedConstructorAccessor228.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1453)
 at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1438)
 at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
 at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:650)
 at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
 at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:531)
 at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:461)
 at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:364)
 at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:317)
 at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:88)
 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
 ... 72 more
Caused by: java.net.SocketTimeoutException: Read timed out
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
 at java.net.SocketInputStream.read(SocketInputStream.java:171)
 at java.net.SocketInputStream.read(SocketInputStream.java:141)
 at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
 at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
 at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1095)
 at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72)
 at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:815)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
 at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
 at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
 at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
 at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
 at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1590)
 at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1516)
 at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1424)
 ... 82 more

When reviewing the Spectrum OC tomcat logs we see a MySql issue. The messages in this instance start with a problem initializing the MySQL connection.

Nov 02, 2021 17:57:24.186 - Could not initialize MySQL connection, retry in 5 seconds: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

After this we see an error about trying to create the netqos_integ database. In this scenario with an migrated OC host we should already have that DB migrated to the new host.

Nov 02, 2021 17:57:25.443 (NETQOS/Startup/Container) (NetqosEventLog) - (ERROR) - Error creating netqos integtration database.
Caused by: com.aprisma.util.mysql.ScriptRunnerException: Error occurred while running SQL script.
  Script: /opt/vgi/CA/Spectrum/Install-Tools/netqos/sql/create_netqos_integ_db.sql
 at com.aprisma.util.mysql.SQLScriptRunner.runScript(SQLScriptRunner.java:162)
 at com.aprisma.util.mysql.SQLScriptRunner.runScript(SQLScriptRunner.java:134)
 at com.ca.im.netqos.integration.dao.jdbc.JdbcSchemaDAO.createDB(JdbcSchemaDAO.java:67)
 at com.ca.im.netqos.integration.dao.jdbc.JdbcSchemaDAO.init(JdbcSchemaDAO.java:47)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:349)
 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:300)
 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:133)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1560)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1471)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1216)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:762)
 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
 at com.ca.spectrum.app.web.spring.BootstrapServlet$ContainerStartupProcess.refreshContainer(BootstrapServlet.java:250)
 at com.ca.spectrum.app.web.spring.BootstrapServlet$ContainerStartupProcess.run(BootstrapServlet.java:208)
 at java.lang.Thread.run(Thread.java:748)
Caused by: com.aprisma.util.mysql.ScriptRunnerException: Error occurred with SQL Script
  Script: /opt/vgi/CA/Spectrum/Install-Tools/netqos/sql/create_netqos_integ_db.sql
  Stdout:
  Stderr:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/vgi/CA/Spectrum/mysql/tmp/mysql.sock' (2)
 at com.aprisma.util.mysql.SQLScriptRunner.doRunSQLScript(SQLScriptRunner.java:251)
 at com.aprisma.util.mysql.SQLScriptRunner.runScript(SQLScriptRunner.java:158)
 ... 34 more

Cause

The Spectrum OC netqos_integ MySql database was not migrated to the new OC host from the old one.

Environment

All supported DX NetOps Performance Management releases

Resolution

Migrate a copy of the netqos_integ database from the old Spectrum OC host to the new Spectrum OC host.

Once migrated restart the OC services and ensure the MySql service runs.

Now edit the Spectrum Data Source setting the new OC server host name and Save the changes which should be successful. After this the Spectrum Data Source should resume successful synchronization with PC Portal.