Agent Upgrade failure from 21.4 to 24.3 & Prod agents not reporting metrics after recycle on 03/18/2025

Products

• DX SaaS

Issue/Introduction

We would like to know what was the RCA and what can be done to avoid such situations. 

 

On 18th March, 2025 we observed multiple issues in our tenant.

1. A Production change ticket for US WebSphere apps’ APM version upgrade failed on 18th March 2025 between 4 and 7 am ET, as the agents (24.3) could not transmit data to the APM via proxy or direct SaaS URL.. 
1. Upon reverting the change to the 21.4 version, the agents started connecting back almost instantly. This agent uses wss to connect to the proxy. 
2. It was also observed that a few other applications were unable to create metrics in the tenant on recycle. These agents were already upgraded to 24.3 in the previous weeks and were using https to connect to the proxy than wss. The apps which failed to create metrics due to recycle were able to do so at around 7pm ET after being disconnected at 5am ET.

 

 

Cause

Wrong connection protocol.

Resolution

The log, 24.3 agent  is using https:
    [Profile:coreProfile                                   ] agentManager.url.1=https://<proxy>:443
    [Profile:coreProfile                                   ] agentManager.url.2=<https://<SAAS_endpoint>:443
 

while the 21.4 agent was using wss:

[INFO}[IntroscopeAgent.Properties] agentManager.url.1=wss://<proxy>:443
[INFO] [IntroscopeAgent.Properties] agentManager.url.2=wss://<SAAS_endpoint>:443
 

It is likely that the agents using https were having an issue at that time.