Blank Page after Login to APM TeamCenter, Status code 503, SSLHandshakeException

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

Symptoms:

- Blank Page after Login
- No data displayed
- Unknown error
- Notifications:

             Error retrieving lazy upgrade status. Status code: undefined
             Error retrieving permissions. Status code: 503
             Error retrieving settings object(s) of the type title-order. Status code 503

- the below exception is reported in the Webview log:

[ERROR] [WebView] The connection with remote resource at https://<em-hostname>:8444/apm/appmap/private/metric/batch has not been established!
[ERROR] [WebView] Unable to establish connection with remote resource at https://<em-hostname>:8444/apm/appmap/private/metric/batch!
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching <em-hostname> found
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
  

Environment

• APM 10.7SP3 or higher
• APM 10.8.x

Cause

Starting from APM 10.7 HF#24 you need to provide a valid ca certificate when using HTTPS, you cannot longer use the self-signed certificate provided by the product's EM installer
 

Resolution

Upload a valid signed certificate to the Enterprise Manager keystore  : <EM_HOME>\config\internal\server\keystore. 

For more information:

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-8/administrating/configure-enterprise-manager/configure-enterprise-manager-communications.html 

Workaround

Create a self-signed certificate with the correct EM server hostname
Below 2 examples explaining the process, for more information about creating a certificate contact your IT Security Team.

In this example:
- EM hostname  = <EM host>
- By default "<EM_HOME>/config/internal/server/keystore" password is "password"
- By default "<EM_HOME>/jre/lib/security/cacerts" password is "changeit"


Example #1: If EM and Webview are installed in the same directory

In this example, introscope is installed in /introscope

Step 1: Backup keystore original files:

/introscope/config/internal/server/keystore,
/introscope/jre/lib/security/cacerts

Step 2: create self-signed certificate

cd /introscope/config/internal/server
"/introscope/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=<EM Host>"
"/introscope/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt 
"/introscope/jre/bin/keytool" -importcert -keystore "/introscope/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit

Step 3: update introscope EM and webview jetty xml files to use the new certificate

By default, Jetty is configured to start a single SSL connector on port 8444. This connector is used for communication with Agents and for HTTP communication over SSL, including Public REST API.
You can reuse the default connector or create a new one, in this example, we are reusing the existing one:

Open /introscope/config/em-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
 <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

Open the /introscope/config/webview-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

NOTE: In case you need to delete the above created jettyssl certificate, you can use the below commands, for more information contact your IT Security team

"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/jre/lib/security/cacerts" -storepass changeit
"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password

Step 4: Start EM and Webview

  
Example #2: If EM and Webview are installed in separate directories

In this example:
- introscope is installed in /introscope
- webview is installed in /introscopeWebview

Step 1: Backup keystore original files:

/introscope/config/internal/server/keystore,
/introscope/jre/lib/security/cacerts
/introscopeWebview/config/internal/server/keystore
/introscopeWebview/jre/lib/security/cacerts

Step 2:  create self-signed certificate

cd /introscope/config/internal/server
"/introscope/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=<EM Host>"
"/introscope/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt 
"/introscope/jre/bin/keytool" -importcert -keystore "/introscope/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit

Step 3: update introscope EM jetty xml files to use the new certificate

By default, Jetty is configured to start a single SSL connector on port 8444. This connector is used for communication with Agents and for HTTP communication over SSL, including Public REST API.
You can reuse the default connector or create a new one, in this example, we are reusing the existing one:

Open /introscope/config/em-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
 <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

Step 4: Import certificate into Webview JRE and update the Webview keystore.

"/introscopeWebview/jre/bin/keytool" -importcert -keystore "/introscopeWebview/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit
cp /introscope/config/internal/server/keystore /introscopeWebview/config/internal/server/keystore


Step 5: update webview jetty xml files to use the new certificate
Open the /introscopeWebview/config/webview-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                   ...

NOTE: In case you need to delete the above created jettyssl certificate, you can use the below commands, for more information contact your IT Security team

"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/jre/lib/security/cacerts" -storepass changeit
"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password
"/introscopeWebview/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscopeWebview/jre/lib/security/cacerts" -storepass changeit
"/introscopeWebview/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscopeWebview/config/internal/server/keystore" -storepass password

Step 6: Start EM and Webview