Unix Agent does not start due to "U02000313 ... Host not found" response

Products

CA Automic Operations Manager CA Automic One Automation CA Automic Workload Automation - Automation Engine Automic SaaS

Issue/Introduction

Unix Agent returns the following response upon agent startup. I determined the root issue is how the system treats the agent INI config file upon startup. When a new version of the INI file is posted to the server, the agent starts with no issues. However, all future instances of the agent restart fails since the system forcibly changed the INI config file format by dropping the domain name from the desired JCP1 value.

Agent log response:

20230127/075035.812 - U02000072 Connection to system 'AUTOMIC' initiated.
20230127/075035.812 - U02000379 Initiating connection to server 'AUTOMIC' using WebSocket URI: 'JCP_SERVER:8443/agent'.
20230127/075036.065 - U02000313 Communication error with partner '*SERVER', error: 'connect/(resolve: Host not found (authoritative) [asio.netdb:1])'.
20230127/075036.065 - U02000010 Connection to Server 'AUTOMIC/unknown' terminated.

Environment

Release : 21.0

Resolution

There are two issues. First we can see in the JCP what hostname is used in the keypair that the JCP keystore uses:

20230114/233755.967 - 34 U00045428 The TLS certificate will expire on: '2023-06-29 01:28:25 UTC'
20230114/233756.014 - 34 Jetty: Logging initialized @116736ms to com.automic.agents.impl.AgentLogBridge
20230114/233756.155 - 34 Jetty: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 11.0.15+10
20230114/233756.483 - 34 Jetty: DefaultSessionIdManager workerName=node0
20230114/233756.483 - 34 Jetty: No SessionScavenger set, using defaults
20230114/233756.498 - 34 Jetty: node0 Scavenging every 660000ms
20230114/233756.811 - 34 Jetty: Started o.e.j.s.ServletContextHandler@7404a9e{/,null,AVAILABLE}
20230114/233757.092 - 34 Jetty: x509=X509@ce49fc7(jetty,h=[JCP_SERVER_NAME.domain.com],w=[]) for Server@347d0366[provider=null,keyStore=file:///C:/automic/AutomationEngine/bin/automic.keystore,trustStore=null]
20230114/233757.155 - 34 Jetty: Started ServerConnector@6452c1d2{SSL, (ssl, http/1.1)}{0.0.0.0:8443}

The agent's JCPlist sows that the AE processes are responding as the servername instead of fully qualified domain name (fqdn):

20230127/075035.810 - [JCPLIST]
20230127/075035.810 - ; List of available JCP endpoints.
20230127/075035.810 - JCP1 = https://JCP_SERVER_NAME:8443/
20230127/075035.810 - JCP2 = https://JCP_SERVER_NAME:8444/

All the AE processes (WPs, CPs, JWPs, JCPs, and REST) are picking up the servername instead of fqdn. I can see this looking at the Host column of the process list in the log file:

20230114/233632.072 - 1 Server Type C Host Port StartTime LastUpdateTime
20230114/233632.087 - 1 ---------------- ---- - ----------------------- ----- ------------------- -------------------
20230114/233632.087 - 1 UC4V21T#CP001 REST 0 JCP_SERVER_NAME 60373 2022-12-22 18:03:27 2023-01-15 04:33:13
20230114/233632.087 - 1 UC4V21T#CP002 JCP 0 JCP_SERVER_NAME 60288 2022-12-22 18:02:06 2023-01-15 04:33:18
20230114/233632.087 - 1 UC4V21T#CP003 JCP * JCP_SERVER_NAME 50002 2023-01-15 04:36:23 2023-01-15 04:36:31
20230114/233632.087 - 1 UC4V21T#WP001 WP 0 JCP_SERVER_NAME 0 2022-12-22 18:00:29 2023-01-15 04:33:15
20230114/233632.087 - 1 UC4V21T#WP002 WP 0 JCP_SERVER_NAME 0 2022-12-22 18:01:00 2023-01-15 04:33:18
20230114/233632.087 - 1 UC4V21T#WP003 WP 0 JCP_SERVER_NAME 0 2023-01-15 04:34:37 2023-01-15 04:36:23
20230114/233632.103 - 1 UC4V21T#WP004 JWP 0 JCP_SERVER_NAME 60250 2022-12-22 18:01:32 2023-01-15 04:33:17
20230114/233632.103 - 1 UC4V21T#WP005 PWP 0 JCP_SERVER_NAME 2270 2023-01-15 04:34:22 2023-01-15 04:36:23
20230114/233632.103 - 1 UC4V21T#WP006 WP 0 JCP_SERVER_NAME 0 2022-12-22 18:00:44 2023-01-15 04:33:19
20230114/233632.103 - 1 UC4V21T#WP007 WP 0 JCP_SERVER_NAME 0 2023-01-15 04:34:53 2023-01-15 04:36:22
20230114/233632.119 - 1 UC4V21T#WP008 WP 0 JCP_SERVER_NAME 0 2023-01-15 04:35:16 2023-01-15 04:36:22

This can be fixed by uncommenting the hostname= setting in ucsrv.ini and using the fully qualified name:

hostName=JCP_SERVER_NAME.domain.com

Then restart the AE processes and they should pick up the fully qualified domain name.

Along with that, although the agent will still connect, we recommend updating the agent ini to use:

connection=JCP_SERVER_NAME.domain.com:8443

The agent will default to the JCPLIST first, but if this behavior changes in the future, the agent won't know what server to reach out to (this is how some older versions worked).

Other notes:

Also note that if there is a space or other character in the JCPs listed under the JCPLIST AND the connection= setting is incorrect, the agnt will not connect correctly to the JCP. The errors may look something like this:

20240715/121640.913 - JCP1 = https:// JCP_SERVERNAME_01:8444/
20240715/121640.913 - JCP2 = https:// JCP_SERVERNAME_01:8445/
...
20240715/121640.913 - U02000379 Initiating connection to server 'AUTOMIC' using WebSocket URI: ' JCP_SERVERNAME_01:8444/agent'.
20240715/121640.914 - U02000313 Communication error with partner '*SERVER', error: 'connect/(resolve: Host not found (authoritative) [asio.netdb:1])'.
20240715/121640.914 - U02000010 Connection to Server 'AUTOMIC/unknown' terminated.
20240715/121640.914 - U02000072 Connection to system 'AUTOMIC' initiated.
20240715/121640.914 - U02000379 Initiating connection to server 'AUTOMIC' using WebSocket URI: ' JCP_SERVERNAME_01:8445/agent'.
20240715/121640.914 - U02000313 Communication error with partner '*SERVER', error: 'connect/(resolve: Host not found (authoritative) [asio.netdb:1])'.
20240715/121640.914 - U02000010 Connection to Server 'AUTOMIC/unknown' terminated.