Products

CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management

Issue/Introduction

This KB article covers an example how to configure APM, Webview to use https and how to configure an instance APMSQLserver remote instance to execute queries using SQuirrel

Environment

Valid for :

Application Performance Management 10.7 SP3 and onward versions

Resolution

Labs details

Server#1 : EM Standalone

CentOS

lvntest004384.test.com

/Introscope1070SP3/

Server#2 : Remove APMSQLServer

CentOS

lvntest010777.test.comt

/APMSQLServer1070SP3-remote

Server#3 : SQuirrel

Windows

ibntest003751.test.com

Configure APM 10.7SP3 (EM + Webview) to use https port 8444 & 8443

This example assumes you have already installed APM 10.7 + applied SP3. You can download package 10.7 and the latest Service Packs from https://support.broadcom.com/download-center/product-download.html?subfamily=APPLICATION%20PERFORMANCE%20MANAGEMENT

1) In this example, we update EM listening port = 7001

Open the EM-HOME/config/IntroscopeEnterpriseManager.properties

introscope.enterprisemanager.port.channel1=7001

2) Enable webstart https connectivity

Open the EM-HOME/config/IntroscopeEnterpriseManager.properties

uncomment

introscope.enterprisemanager.webserver.jetty.configurationFile=em-jetty-config.xml

This change instruct the EM to use by default secure port 8444 port defined in em-jetty-config.xml

...

</Array>

</Arg>

</New>

</Arg>

</Call>

<!-- Configure non-secure http connector for the Jetty Server

<Arg>

..

3) update Webview settings

a) Enable webview https connectivity

Open the EM-HOME/config/IntroscopeWebview.properties

uncomment

#introscope.webview.jetty.configurationFile=webview-jetty-config.xml

This change instruct the EM to use by default secure port 8443 defined in webview-jetty-config.xml

..

</Array>

</Arg>

</New>

</Arg>

</Call>

<!-- Configure non-secure http connector

b) Update Webview to EM connection

introscope.webview.enterprisemanager.tcp.host=lvntest004384.test.com

introscope.webview.enterprisemanager.tcp.port=7001

c) Update EM webstart port + protocol

introscope.webview.enterprisemanager.webserver.tcp.port=8444

# URL that points to the root of the Enterprise Manager REST API

introscope.webview.enterprisemanager.rest.base=https://lvntest004384.test.com:8444/apm/appmap

d) Start EM and Webview

e) Verify for possible errors in EM and Webview logs.

No errors, all seems to be correct

f) connect to APM TeamCenter, you encounter below error:

PROBLEM#1: Error “Status Code: 503” when connecting to ATC, empty page

In Webview log:

12/01/20 05:12:59.549 PM UTC [INFO] [WebView.Login] Successfully logged in user "WilyWebView"

12/01/20 05:12:59.592 PM UTC [INFO] [WebServer] Web Application Server started

12/01/20 05:15:03.755 PM UTC [INFO] [com.wily.introscope.webserver] Login event Admin 10.230.40.46 true

12/01/20 05:15:05.621 PM UTC [ERROR] [WebView] Unable to establish connection with remote resource at https://lvntest004384.test.com:8444/apm/appmap/private/configuration/static!

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching lvntest004384.test.com found

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)

This is a know issue documented in this KB:

https://knowledge.broadcom.com/external/article?articleId=125671

Root cause :

You cannot longer use the self-signed certificate provided by the product's EM installer

Solution: Workaround: configure EM to use a self-signed certificate as suggested in above KB 125671

https://knowledge.broadcom.com/external/article?articleId=125671

Below detail of the steps (using KB as template) for this test setup

Step 1: skipped

Step 2: create self-signed certificate

[[email protected] bin]# cd /Introscope1070SP3/config/internal/server

[[email protected] server]# "/Introscope1070SP3/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/Introscope1070SP3/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=lvntest004384.test.com"

[[email protected] server]# "/Introscope1070SP3/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt

Certificate stored in file <jettyssl.crt>

[[email protected] server]# "/Introscope1070SP3/jre/bin/keytool" -importcert -keystore "/Introscope1070SP3/jre/lib/security/cacerts" -alias jettyssl -file "/Introscope1070SP3/config/internal/server/jettyssl.crt" -storepass changeit

Trust this certificate? [no]: yes

Certificate was added to keystore

Step 3: update introscope EM and webview jetty xml files to use the new certificate

<Set name="certAlias">jettyssl</Set>

Step 4: Start EM and Webview

You should now be able to successfully access ATC

Deploy Remote APMSQL Server instance

Install APMSQLServer remotely as per documentation:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/installing/apm-installation/install-and-configure-apmsql-servers.html#concept.dita_9fb6774b89063667339f1895a24a4cf99fa9e2cf_InstallAPMSQLServeronaRemoteServer

Step by Step instructions:

a) Go to the remote server, create a home directory.

In this example: /APMSQLServer1070SP3-remote

b) Install Java, in this example jdk-8u231-linux-x64.tar.gz

copy the file to /APMSQLServer1070SP3-remote/

tar xvf jdk-8u231-linux-x64.tar.gz

export JAVA_HOME=/APMSQLServer1070SP3-remote/jdk1.8.0_231

c) As per documentation, create a config directory:

mkdir /APMSQLServer1070SP3-remote/config

d) As per documentation, copy <EM-HOME>/APMSQLServer directory to the remote server, in this example “lvntest010777”, to the target directory /APMSQLServer1070SP3-remote

scp -r /Introscope1070SP3/APMSqlServer/ [email protected]:/APMSQLServer1070SP3-remote

e) As per documentation, copy <EM-HOME>\config\APMSqlServer.properties directory to the config directory

scp -r /Introscope1070SP3/config/APMSqlServer.properties [email protected]:/APMSQLServer1070SP3-remote/config

f) Verification: Check the content of remote APMSQLServer, in this example @lvntest010777/APMSQLServer1070SP3-remote:

[[email protected] APMSQLServer1070SP3-remote]# ls -l

total 189604

drwxr-xr-x. 9 root root 95 Dec 1 16:52 APMSqlServer

drwxr-xr-x. 2 root root 37 Dec 1 22:26 config

drwxr-xr-x. 7 10 143 245 Oct 5 2019 jdk1.8.0_231

-rw-r--r--. 1 root root 194151339 Dec 1 22:13 jdk-8u231-linux-x64.tar.gz

Configure APMSQL Server with remote EM

Summary of servers and ports:

EM = lvntest004384.test.com

EM https webport = 8444

APMSQLServer = lvntest010777.test.com

a) Open /APMSQLServer1070SP3-remote/config/APMSqlServer.properties, update below properties, for this example:

# The EM or MOM (hostname or IP address) to which APMSQLServer connects.

# Default is localhost.

com.ca.apm.sqlserver.em.host=lvntest004384.test.com

# The EM or MOM webserver port.

# Default is 7081.

# Should have the same value as 'introscope.enterprisemanager.webserver.port' property defined

# in IntroscopeEnterpriseManager.properties.

com.ca.apm.sqlserver.em.webserver.port=8444

# Jdbc bind address using which the client API will connect to APMSQLServer.

# Default is localhost.

# Make sure to use IP address/host name for JDBC bind address.

# Also, use the same IP address/host name in the client to connect.

com.ca.apm.sqlserver.jdbcbind.address=lvntest010777.test.com

..

com.ca.apm.sqlserver.em.webserver.connection.protocol=https

Enabled DEBUG logging (just for verification, once all is working fine, you can disable it)

-Open /config/APMSqlServer.properties

-Set

log4j.rootLogger=ALL,console

log4j.logger.org.teiid=ALL,teiidlog

log4j.logger.com.ca.apm.server=ALL,console,logfile

Save the properties file

b) start apmsqlserver

cd /APMSQLServer1070SP3-remote/APMSqlServer/bin

./apmsql

c) Verify APMSQLServer logs:

open /APMSQLServer1070SP3-remote/APMSqlServer/logs\apmsqlserverout.log

Go the end of the log, verify that server has started successfully

020-12-01 22:27:02.398:INFO:oejs.Server:main: Started @37879ms

Install SQuirrel + Setup APM driver

In Windows server:

1.Download and install Squirrel SQL Client : http://www.squirrelsql.org/

2.Copy teiid-9.0.1-jdbc.jar from APMSQLServer : /APMSQLServer1070SP3-remote/APMSqlServer/client/teiid-9.0.1-jdbc.jar

to the local windows server, for example D:

3.Open Squirrel

4. Create a new Driver

a) Click Drivers,

b) Click “+” icon:

c) In the “Add Driver” window, set

Name = TeiidDriver_sp3
Example URL = jdbc:teiid:[email protected]://test.com:54321

d) Click “Extra Class Path” tab

e) Click “Add” and browse and select D:\teiid\teiid-9.0.1-jdbc.jar

Set “Class Name” = org.teiid.jdbc.TeiidDriver

f) Click OK

5.Create a new Alias

Click Aliases

Click “+” icon

Set

-Name = apmsqlserver - lvntest010777
-Driver = locate TeiidDriver you created in previous step

-URL

jdbc:teiid:[email protected]://lvntest010777.test.com:54321

-User Name and Password : you can use the below options:

a) Enter the user/password

b) Enter a Public API token created from Team Center (go to Settings > Security> Click Generate Public API Token).

IMPORTANT: You must create a token using an “Admin” account to prevent some known issue, for example: https://knowledge.broadcom.com/external/article?articleId=74770

NOTE: In his example, we use option “a”, we have created admin user “test/test” in <EM-home>/config/users.xml

Click “Test”

TEST#1: Connect to APMSQLServer using an Admin user or Tolken without having imported certificate in remote jvm

Result:

We encounter the below error/exception:

Error:

Unexpected Error occurred attempting to open an SQL connection.

class sun.security.provider.certpath.SunCertPathBuilderException: Remote sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Stack Trace

sun.security.provider.certpath.SunCertPathBuilderException: Remote sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)

at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)

at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)

at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)

at sun.security.validator.Validator.validate(Validator.java:262)

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1622)

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)

at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)

at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)

at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)

at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)

at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:275)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1345)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1306)

at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:307)

at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)

at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69)

at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1358)

at org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:77)

at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)

at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:673)

at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)

at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:710)

at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1050)

at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897)

at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:866)

at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:334)

at org.apache.cxf.jaxrs.client.WebClient.post(WebClient.java:343)

at com.ca.apm.server.teiid.APMSqlSecurityHelper.isValid(APMSqlSecurityHelper.java:199)

at com.ca.apm.server.teiid.APMSqlSecurityHelper.authenticate(APMSqlSecurityHelper.java:128)

at com.ca.apm.server.teiid.APMSqlSecurityHelper.authenticate(APMSqlSecurityHelper.java:1)

at org.teiid.services.SessionServiceImpl.createSession(SessionServiceImpl.java:206)

at org.teiid.transport.LogonImpl.logon(LogonImpl.java:142)

at org.teiid.transport.LogonImpl.logon(LogonImpl.java:127)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.teiid.transport.ServerWorkItem.run(ServerWorkItem.java:87)

at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276)

at org.teiid.transport.SocketClientInstance.processMessagePacket(SocketClientInstance.java:236)

at org.teiid.transport.SocketClientInstance.receivedMessage(SocketClientInstance.java:222)

at org.teiid.transport.SSLAwareChannelHandler.messageReceived(SSLAwareChannelHandler.java:212)

at org.teiid.transport.SSLAwareChannelHandler.channelRead(SSLAwareChannelHandler.java:218)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)

at java.lang.Thread.run(Thread.java:748)

- Review the log /APMSqlServer/logs/apmsqlserverout.log, you will find details of exception

12/04/20 09:54:26.942 AM SAST [TRACE] [NIO2] [APMSQLServer] endpoint url :https://lvntest004384.test.com:8444/apm/appmap/private/token/temporaryToken

Dec 04, 2020 9:54:27 AM org.apache.cxf.phase.PhaseInterceptorChain doDefaultLogging

WARNING: Interceptor for {https://lvntest004384.test.com:8444/apm/appmap/private/token/temporaryToken}WebClient has thrown exception, unwinding now

org.apache.cxf.interceptor.Fault: Could not send Message.

at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:67)