Spectrum Data Source sync failure post migration
Article ID: 227693
Updated On:
Products
Issue/Introduction
The PC Portal and Spectrum OC hosts were migrated to new hosts with new names.
Everything is working using new SSL certificates with the new host names.
The OC SSL root and intermediate certificates were exported from OC and imported to PC Portal.
Trying to Save the Spectrum Data Source in the Portal Web UI after setting it's new host name fails. Sync is also failing.
When trying to Save the Spectrum Data Source in PC Portal it failed with this PCService.log error. It indicates an inability to successfully communicate with the Spectrum OC host. It suggests review of the Spectrum OC host tomcat logs is the next step.
ERROR | qtp1093605336-1770 | 2021-11-02 19:21:08,561 | com.ca.im.portal.site.admin.models.DataSourceAdminModel
| Failed to create datasource: Could not send Message.
javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:145)
at com.sun.proxy.$Proxy114.productID(Unknown Source)
at com.ca.im.portal.api.services.datasource.ProductSync2WSProxyImpl.getProductId(ProductSync2WSProxyImpl.java:43)
at com.ca.im.portal.api.services.datasource.DataSourcePoll.getDataSourceGuid(DataSourcePoll.java:419)
at com.ca.im.portal.api.services.datasource.DataSourcePoll.pollProductSyncWS(DataSourcePoll.java:140)
at com.ca.im.portal.api.services.datasource.DataSourcePoll.poll(DataSourcePoll.java:98)
at com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl.buildImpl(DataSourceBuilderImpl.java:149)
at com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl.build(DataSourceBuilderImpl.java:125)
at com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl.commitWithId(DataSourceBuilderImpl.java:187)
at com.ca.im.portal.site.admin.models.DataSourceAdminModel.create(DataSourceAdminModel.java:655)
at com.ca.im.portal.site.admin.models.DataSourceAdminModel.handleFormSubmit(DataSourceAdminModel.java:934)
at com.ca.im.portal.render.desktop.report.FormRenderer.processFormSubmit(FormRenderer.java:131)
at com.ca.im.portal.render.desktop.report.FormRenderer.requestFromClient(FormRenderer.java:99)
at com.ca.im.portal.render.desktop.BaseRenderer.handleRequestFromClient(BaseRenderer.java:398)
at com.ca.im.portal.render.desktop.DesktopRenderMgr.processClientRequest(DesktopRenderMgr.java:323)
at com.ca.im.portal.desktop.PageData.doPost(PageData.java:190)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.ca.im.portal.common.web.security.RequestFilter.doFilter(RequestFilter.java:27)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.ca.im.portal.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking https://<Spectrum_OC_Host>:8443/axis2/services/ProductSync2WS: Read timed out
at sun.reflect.GeneratedConstructorAccessor228.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1453)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1438)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:650)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:531)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:461)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:364)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:317)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:88)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
... 72 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1095)
at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:815)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1590)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1516)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1424)
... 82 more
When reviewing the Spectrum OC tomcat logs we see a MySql issue. The messages in this instance start with a problem initializing the MySQL connection.
Nov 02, 2021 17:57:24.186 - Could not initialize MySQL connection, retry in 5 seconds: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
After this we see an error about trying to create the netqos_integ database. In this scenario with an migrated OC host we should already have that DB migrated to the new host.
Nov 02, 2021 17:57:25.443 (NETQOS/Startup/Container) (NetqosEventLog) - (ERROR) - Error creating netqos integtration database.
Caused by: com.aprisma.util.mysql.ScriptRunnerException: Error occurred while running SQL script.
Script: /opt/vgi/CA/Spectrum/Install-Tools/netqos/sql/create_netqos_integ_db.sql
at com.aprisma.util.mysql.SQLScriptRunner.runScript(SQLScriptRunner.java:162)
at com.aprisma.util.mysql.SQLScriptRunner.runScript(SQLScriptRunner.java:134)
at com.ca.im.netqos.integration.dao.jdbc.JdbcSchemaDAO.createDB(JdbcSchemaDAO.java:67)
at com.ca.im.netqos.integration.dao.jdbc.JdbcSchemaDAO.init(JdbcSchemaDAO.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:349)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:300)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:133)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1560)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1471)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1216)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:762)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at com.ca.spectrum.app.web.spring.BootstrapServlet$ContainerStartupProcess.refreshContainer(BootstrapServlet.java:250)
at com.ca.spectrum.app.web.spring.BootstrapServlet$ContainerStartupProcess.run(BootstrapServlet.java:208)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.aprisma.util.mysql.ScriptRunnerException: Error occurred with SQL Script
Script: /opt/vgi/CA/Spectrum/Install-Tools/netqos/sql/create_netqos_integ_db.sql
Stdout:
Stderr:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/vgi/CA/Spectrum/mysql/tmp/mysql.sock' (2)
at com.aprisma.util.mysql.SQLScriptRunner.doRunSQLScript(SQLScriptRunner.java:251)
at com.aprisma.util.mysql.SQLScriptRunner.runScript(SQLScriptRunner.java:158)
... 34 more
Environment
All supported DX NetOps Performance Management releases
Cause
The Spectrum OC netqos_integ MySql database was not migrated to the new OC host from the old one.
Resolution
Migrate a copy of the netqos_integ database from the old Spectrum OC host to the new Spectrum OC host.
Once migrated restart the OC services and ensure the MySql service runs.
Now edit the Spectrum Data Source setting the new OC server host name and Save the changes which should be successful. After this the Spectrum Data Source should resume successful synchronization with PC Portal.
Feedback
