AuthHub ssp chart helm installation failed due to dataseed job BackoffLimitExceeded error
Article ID: 273180
Updated On:
Products
Issue/Introduction
We are getting we below error while deploying Vip AuthHub on EKS 1.23. SSP installation is failing with error
“Error: failed pre-install: job failed: BackoffLimitExceeded”
and ssp-ssp-dataseed pod failed to initialize for VIP AuthHub ver 2.1.4
EKS version: 1.23
ssp_version="2.1.4+1042"
ssp-infra="2.1.4+1042"
ssp_data_version="2.1.0+1147"
ssp-sample-app="2.1.4+1042"
The following shows the dataseed job pod's status has Init:Error
kubectl get pod -n ssp
NAME READY STATUS RESTARTS AGE
ssp-ssp-dataseed-job-5kxdt 0/1 Init:Error 0 110s
The following events are shown running
kubectl describe job ssp-ssp-dataseed-job -n ssp
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 5m40s job-controller Created pod: ssp-ssp-dataseed-job-qpw6v
Warning BackoffLimitExceeded 4m10s job-controller Job has reached the specified backoff limit
Environment
Release : VIP Authentication Hub 2.1.x
Cause
AWS Security Group (firewall) blocks ssp dataseed job container from accessing the external database
Resolution
We run
kubectl logs <ssp-dataseed-job pod> -n ssp -c ssp-db-init > ssp-db-init.log
From ssp-db-init.log we found there was connection timeout when the ssp-db-init container attempted to connect via jdbc to (host=xxxx.xxx.aws.au.xxx.net)(port=3306)
2023-09-05T17:54:20.420Z ERROR 1 --- [ main] c.z.h.p.HikariPool : spring.write.datasource - Exception during pool initialization.
java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=xxxx.xxx.aws.au.xxx.net)(port=3306)(type=master) : Socket fail to connect to host:xxxx.xxx.aws.au.xxx.net, port:3306. Connect timed out
...
Caused by: java.sql.SQLNonTransientConnectionException: Socket fail to connect to host:xxxx.xxx.aws.au.xxx.net, port:3306. Connect timed out
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73) ~[mariadb-java-client-2.7.3.jar!/:?]
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:185) ~[mariadb-java-client-2.7.3.jar!/:?]
...
Caused by: java.net.SocketTimeoutException: Connect timed out
at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
at sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) ~[?:?]
This is due to AWS Security Group (firewall) settings that blocks the access from the ssp db init container to external database. Customer fixes the AWS Security Group configuration to address this issue.
Feedback
