Team Center - Blank Page after Login, Status code 503, SSLHandshakeException

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Symptoms:

- Blank Page after Login

- No data displayed

- Unknown error

- Notifications:

             Error retrieving lazy upgrade status. Status code: undefined
             Error retrieving permissions. Status code: 503
             Error retrieving settings object(s) of the type title-order. Status code 503

- the below exception is reported in the Webview log:

[ERROR] [WebView] The connection with remote resource at https://<em-hostname>:8444/apm/appmap/private/metric/batch has not been established!
[ERROR] [WebView] Unable to establish connection with remote resource at https://<em-hostname>:8444/apm/appmap/private/metric/batch!
javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching <em-hostname> found
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
    

Environment

Applies ONLY to environments using the below configuration:

- APM 10.7 Hotfix # 24 or any higher version (including SP3)
- You are using HTTPS to connect to Webview
- You are using HTTPS to connect to EM REST API (introscope.webview.enterprisemanager.rest.base=https ... in IntroscopeWebView.properties)
- You are using the default self-signed certificate installed by the product's EM installer and WebView
 

Cause

Starting from APM 10.7 HF#24 you need to provide a valid certificate when using HTTPS
You cannot longer use the self-signed certificate provided by the product's EM installer
 

Resolution

Upload a valid signed certificate to the Enterprise Manager keystore  : <EM_HOME>\config\internal\server\keystore. 

For more information:

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/administrating/configure-enterprise-manager/configure-enterprise-manager-communications.html#concept.dita_ae08e8af60230e54a160dd28610d324e88718ff1_ConfigureEnterpriseManagertoWebViewCommunicationsSSL

Workaround

Create a self-signed certificate with the correct EM server hostname
Below 2 examples explaining the process, for more information about creating a certificate contact your IT Security Team.

In this example:
- EM hostname  = <EM host>
- By default "<EM_HOME>/config/internal/server/keystore" password is "password"
- By default "<EM_HOME>/jre/lib/security/cacerts" password is "changeit"


Example #1: If EM and Webview are installed in the same directory

In this example, introscope is installed in /introscope

Step 1: Backup keystore original files:

/introscope/config/internal/server/keystore,
/introscope/jre/lib/security/cacerts

Step 2: create self-signed certificate

cd /introscope/config/internal/server
"/introscope/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=<EM Host>"
"/introscope/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt 
"/introscope/jre/bin/keytool" -importcert -keystore "/introscope/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit

Step 3: update introscope EM and webview jetty xml files to use the new certificate

By default, Jetty is configured to start a single SSL connector on port 8444. This connector is used for communication with Agents and for HTTP communication over SSL, including Public REST API.
You can reuse the default connector or create a new one, in this example, we are reusing the existing one:

Open /introscope/config/em-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
 <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

Open the /introscope/config/webview-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

NOTE: In case you need to delete the above created jettyssl certificate, you can use the below commands, for more information contact your IT Security team

"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/jre/lib/security/cacerts" -storepass changeit
"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password

Step 4: Start EM and Webview

  
Example #2: If EM and Webview are installed in separate directories

In this example:
- introscope is installed in /introscope
- webview is installed in /introscopeWebview

Step 1: Backup keystore original files:

/introscope/config/internal/server/keystore,
/introscope/jre/lib/security/cacerts
/introscopeWebview/config/internal/server/keystore
/introscopeWebview/jre/lib/security/cacerts

Step 2:  create self-signed certificate

cd /introscope/config/internal/server
"/introscope/jre/bin/keytool" -genkey -keyalg RSA -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password -keypass password -validity 7300 -dname "CN=<EM Host>"
"/introscope/jre/bin/keytool" -export -alias jettyssl -keystore keystore -storepass password -file jettyssl.crt 
"/introscope/jre/bin/keytool" -importcert -keystore "/introscope/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit

Step 3: update introscope EM jetty xml files to use the new certificate

By default, Jetty is configured to start a single SSL connector on port 8444. This connector is used for communication with Agents and for HTTP communication over SSL, including Public REST API.
You can reuse the default connector or create a new one, in this example, we are reusing the existing one:

Open /introscope/config/em-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
 <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty name="introscope.config" default="./config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                    ...

Step 4: Import certificate into Webview JRE and update the Webview keystore.

"/introscopeWebview/jre/bin/keytool" -importcert -keystore "/introscopeWebview/jre/lib/security/cacerts" -alias jettyssl -file "/introscope/config/internal/server/jettyssl.crt" -storepass changeit
cp /introscope/config/internal/server/keystore /introscopeWebview/config/internal/server/keystore


Step 5: update webview jetty xml files to use the new certificate
Open the /introscopeWebview/config/webview-jetty-config.xml and update certAlias, replace caapm with jettyssl
....
  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ServerConnector">
        <Arg name="server">
          <Ref refid="Server"/>
        </Arg>
        <Arg name="factories">
          <Array type="org.eclipse.jetty.server.ConnectionFactory">
            <Item>
              <New class="org.eclipse.jetty.server.SslConnectionFactory">
                <Arg name="next">http/1.1</Arg>
                <Arg name="sslContextFactory">
                  <New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
                    <Set name="KeyStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="KeyStorePassword"><password></Set>
                    <Set name="TrustStorePath">
                      <SystemProperty default="./config" name="introscope.config"/>/internal/server/keystore
                    </Set>
                    <Set name="TrustStorePassword"><password></Set>
                    <Set name="certAlias">jettyssl</Set>
                    <Set name="validateCerts">false</Set>
                    <Set name="needClientAuth">false</Set>
                   ...

NOTE: In case you need to delete the above created jettyssl certificate, you can use the below commands, for more information contact your IT Security team

"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/jre/lib/security/cacerts" -storepass changeit
"/introscope/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscope/config/internal/server/keystore" -storepass password
"/introscopeWebview/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscopeWebview/jre/lib/security/cacerts" -storepass changeit
"/introscopeWebview/jre/bin/keytool" -delete -alias jettyssl -keystore "/introscopeWebview/config/internal/server/keystore" -storepass password

Step 6: Start EM and Webview



NOTES:

1. <Set name="verifyHostnames">  is not longer available in the upgraded jetty version, after the HOTFIX is installed, this property will be removed from the jetty xml file
Do not add the property, otherwise, the below exception will be reported in the EM log during EM startup:

1/29/19 04:20:23.437 PM EST [ERROR] [main] [Manager.EMWebServer] Error loading /introscope/./config/em-jetty-config.xml
org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration$1: class org.eclipse.jetty.util.ssl.SslContextFactory.setVerifyHostnames(class java.lang.String)
    at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.set(XmlConfiguration.java:661)
    
2. If during EM startup you notice the below exception in the EM log, you must reenter the credentials in the EM-HOME\config\tess-db-cfg.xml:

1/29/19 04:23:21.184 PM EST [ERROR] [main] [org.springframework.web.context.ContextLoader] Context initialization failed
com.ca.apm.crypto.InvalidConfigurationException: Given final block not properly padded

Open the EM-HOME/config/tess-db-cfg.xml
Update "plainTextPasswords" to false  and reenter the "hibernate.connection.password"
Restart the EM 

Additional Information

https://knowledge.broadcom.com/external/article?articleId=105898

Note :This exact issue can happen also for non-SSL URLs. One cause may be that the APM Database assisted triage tables are very large and need cleanup. 

Important Note: 

This issue could be due to a Java Upgrade on EM or Webview and the cacerts (Java Keystore) is missing the EM or Webview Certificate.