Fault Tolerant Data Aggregator fails to start after 25.4.2 upgrade
search cancel

Fault Tolerant Data Aggregator fails to start after 25.4.2 upgrade

book

Article ID: 418125

calendar_today

Updated On:

Products

Network Observability CA Performance Management

Issue/Introduction

Upgraded Fault Tolerant (FT) Data Aggregator (DA) servers. Upgraded the proxy, one DA, then the second DA, in the documented order.

After the upgrade both DA's fail to start.

The consul service on both DA servers shows as active and running. Despite this when we run "systemctl status consul" we see the following errors.

Nov 11 10:52:05 <DAHostName> consul[1485]: 2025-11-11T10:52:05.632-0700 [WARN]  agent: Check is now critical: check=service:daservice
Nov 11 10:52:06 <DAHostName> consul[1485]: 2025-11-11T10:52:06.403-0700 [WARN]  agent: Coordinate update blocked by ACLs: accessorID="anonymous token"
Nov 11 10:52:15 <DAHostName> consul[1485]: 2025-11-11T10:52:15.634-0700 [WARN]  agent: Check is now critical: check=service:daservice
Nov 11 10:52:16 <DAHostName> consul[1485]: 2025-11-11T10:52:16.908-0700 [ERROR] agent.http: Request error: method=PUT url=/v1/session/renew/24ea144c-715c-####-####-f4d4530f90d8 from=127.0.0.1:39672 error="Session id '24ea144c-715c-####-####-f4d4530f90d8' not found"
Nov 11 10:52:24 <DAHostName> consul[1485]: 2025-11-11T10:52:24.474-0700 [WARN]  agent: [core][Channel #1 SubChannel #12]grpc: addrConn.createTransport failed to connect to {Addr: "proxyHost:8300", ServerName: "<ProxyHostName", }. Err: connection error: desc = "transport: Error while dialing: dial tcp DA_IP:0->Proxy_IP:8300: operation was canceled"

The consul-ext.log file shows the following error.

ERROR | healthtaskmanager-task-1 | 2025-11-11 10:43:06,279 | com.ca.consul.ext.executor.GroovyExecutor                        
      | groovy execution failure: /opt/ca/IMDataAggregator/consul-ext/scripts/startDA.groovy
com.ca.consul.ext.executor.GroovyExecutionException: (Error Line: 26)
Exception java.lang.UnsatisfiedLinkError: /tmp/bc-fips-jni_2249288415743907/libbc-probe.so: /tmp/bc-fips-jni_2249288415743907/libbc-probe.so: failed to map segment from shared object [in thread "healthtaskmanager-task-1"]
...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.bouncycastle.crypto.CryptoServicesRegistrar
...
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/bc-fips-jni_2249288415743907/libbc-probe.so: /tmp/bc-fips-jni_2249288415743907/libbc-probe.so: failed to map segment from shared object [in thread "healthtaskmanager-task-1"]

Environment

All supported Network Observability DX NetOps Performance Management Fault Tolerant Data Aggregator clusters

Cause

FT DA Consul upgrade on both DA's and the proxy host where the servers are configured with:

  • /tmp set to noexec
  • fips is enabled

Resolution

To resolve this we'll need to redirect the consul services to utilize an alternative tmp directory. To do this make the following changes.

NOTE: Modify the script in the same file on both DA servers.

Default paths are shown.

  1. Open the /opt/IMDataAggregator/consul-ext/bin/start-consul-ext.sh script file for editing.
    • Find the following default entry for JAVA_OPTS that shows the following.
      • JAVA_OPTS="-Dlog4j2.configurationFile=file:$SERVICE_DIR/conf/log4j2.properties"
  2. Modify the entry by adding the bolded text to the line. We're adding only, not removing anything.
      • JAVA_OPTS="-Djava.io.tmpdir=/opt/java-tmp -Dlog4j2.configurationFile=file:$SERVICE_DIR/conf/log4j2.properties"
  3. Save the changes to the file on both DA hosts.
  4. Stop the running consul-ext service on both DA's and the proxy host.
  5. Start consul-ext on both DA hosts.
  6. Activate the DAs.
Powered by Wolken Software