Hazelcast pods not coming up after enabling Dynatrace monitoring

Products

• VIP Authentication Hub

Issue/Introduction

Hazelcast pods fails to start after Dynatrace is enabled.

Below Error can be seen from the failing POD 

$ k logs -f ssp-hazelcast-0                                                                                                                                              
<path>/hazelcast/bin/common.sh: line 38: [: : integer expression expected                                                                                                  
########################################                                                                                                                                 
# JAVA=<path>/java/openjdk/bin/java                                                                                                                                        
# JAVA_OPTS=-Dhazelcast.logging.type=log4j2 -Dlog4j.configurationFile=file:<path>/hazelcast/config/log4j2.properties -Dhazelcast.config=<path>/hazelcast/config/hazelcast-doc
ker.xml -Djet.custom.lib.dir=<path>/hazelcast/custom-lib -Djava.net.preferIPv4Stack=true -XX:MaxRAMPercentage=80.0 -XX:MaxGCPauseMillis=5 -Dhazelcast.config=/data/hazelcas
t/hazelcast.yaml -DserviceName=ssp-hazelcast -Dnamespace=ssp -Dhazelcast.hotRestart=false -Dhz.jet.enabled=true -Dhazelcast.shutdownhook.policy=GRACEFUL -Dhazelcast.shut
downhook.enabled=true -Dhazelcast.graceful.shutdown.max.wait=600 -Dhazelcast.cluster.version.auto.upgrade.enabled=true                  

Environment

Any SSP Release prior of 2.2.1 

Cause

The ssp-hazelcast pod in Authentication Hub encountered startup failures when the third-party component, Dynatrace (OneAgent), was enabled.

This occurred due to a compatibility issue between ssp-hazelcast pod and Dynatrace, triggered by the auto-injection of third-party libraries

Resolution

Couple of things you can do to resolve the issue 

** Option 1) -->  Disable Dynatrace Injections into the Hazelcast POD.

From Dynatrace to completely disable all pod modifications via webhook, set all three annotations to false.

- data-ingest.dynatrace.com/inject: false
- oneagent.dynatrace.com/inject: false
- dynatrace.com/inject: false

Reference to Dynatrace document -->  https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-k8s/guides/operation/annotate#pod

** Option 2 ) -->  Upgrade to SSP release 2.2.1 or higher where the libraries compatibility issues with Dynatrace and Hazelcast has been addressed.