OTK AWS/Kubernetes install issue
search cancel

OTK AWS/Kubernetes install issue

book

Article ID: 406823

calendar_today

Updated On:

Products

CA API Gateway

Issue/Introduction

The Gateway and the OTK chart seems to be properly install :

jenkins@cfbb8e1b9489:~$ helm install my-ssg layer7/gateway --set-file "license.value=./layer7/license_v11.xml" --set "license.accept=true" -f ./layer7/my_values.yaml --timeout 900s
NAME: my-ssg
LAST DEPLOYED: Wed Aug  6 07:58:07 2025
NAMESPACE: layer7-dev-poc
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
##################################################################################
####                                 Success!                                 ####
##################################################################################
####                 Your gateway deployment has been INSTALLED               ####
##################################################################################

To view the Gateway's services you can use the following command
$ kubectl get svc -n layer7-dev-poc | grep gateway

You configured the following ingress hosts
- poclayer7.emea.caas.oneadp.com
- poclayer7.pm.emea.caas.oneadp.com

To learn more about the Gateway Helm Chart check out the following links

Gateway Helm Chart Readme
- https://github.com/CAAPIM/apim-charts/tree/stable/charts/gateway

Thinking in Kubernetes
- https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-gateway/congw-10-1/learning-center/thinking-in-kubernetes.html#thinkingk8s


but the result of kubectl get all command show the following:

jenkins@cfbb8e1b9489:~$ kubectl get all
NAME                                           READY   STATUS      RESTARTS   AGE
pod/my-ssg-gateway-795fdb97f5-lg8zw            1/1     Running     0          4m20s
pod/my-ssg-gateway-otk-db-upgrade-s49fd        0/1     Completed   0          4m51s
pod/my-ssg-gateway-otk-install-8lgrh           0/1     Error       0          4m19s
pod/my-ssg-gateway-otk-install-nzttj           0/1     Completed   0          37s
pod/my-ssg-gateway-pm-tagger-765bcdbbd-lmndx   1/1     Running     0          4m20s

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/my-ssg-gateway              ClusterIP   172.20.202.22   <none>        8443/TCP   4m21s
service/my-ssg-gateway-management   ClusterIP   172.20.10.84    <none>        9443/TCP   4m21s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-ssg-gateway             1/1     1            1           4m21s
deployment.apps/my-ssg-gateway-pm-tagger   1/1     1            1           4m21s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/my-ssg-gateway-795fdb97f5            1         1         1       4m21s
replicaset.apps/my-ssg-gateway-pm-tagger-765bcdbbd   1         1         1       4m21s

NAME                                      STATUS     COMPLETIONS   DURATION   AGE
job.batch/my-ssg-gateway-otk-db-upgrade   Complete   1/1           32s        4m53s
job.batch/my-ssg-gateway-otk-install      Complete   1/1           3m46s      4m20s

 

Detail log of the pod in error state:


jenkins@cfbb8e1b9489:~$ kubectl logs my-ssg-gateway-otk-install-8lgrh
OTK_DATABASE_UPGRADE :  false
OTK_SK_UPGRADE :  true
OTK_SCHEDULE_JOB_SERVICE :
2025-08-06 07:58:49,722 - root - INFO - ****************************************************
2025-08-06 07:58:49,722 - root - INFO - ****************************************************
2025-08-06 07:58:49,722 - root - INFO - ****************************************************
2025-08-06 07:58:49,722 - root - INFO - OTK Install Mode                     : job
2025-08-06 07:58:49,722 - root - INFO - OTK Version                          : 4.6.4-8522
2025-08-06 07:58:49,722 - root - INFO - OTK force Install/Upgrade            : false
2025-08-06 07:58:49,722 - root - INFO - OTK type                             : SINGLE
2025-08-06 07:58:49,722 - root - INFO - DB type                              : mysql
2025-08-06 07:58:49,722 - root - INFO - Portal integration                   : false
2025-08-06 07:58:49,722 - root - INFO - Skip post installation tasks         : false
2025-08-06 07:58:49,722 - root - INFO - Restman Host                         : my-ssg-gateway
2025-08-06 07:58:49,722 - root - INFO - Restman Port                         : 8443
2025-08-06 07:58:49,722 - root - INFO - Gateway Admin User                   : admin
2025-08-06 07:58:49,722 - root - INFO - Internal Gateway Host                :
2025-08-06 07:58:49,722 - root - INFO - Internal Gateway Port                :
2025-08-06 07:58:49,722 - root - INFO - DMZ Gateway Host                     :
2025-08-06 07:58:49,722 - root - INFO - DMZ Gateway Port                     :
2025-08-06 07:58:49,722 - root - INFO - DMZ Gateway Cert                     : None
2025-08-06 07:58:49,722 - root - INFO - DMZ Gateway Cert Issuer              : None
2025-08-06 07:58:49,722 - root - INFO - DMZ Gateway Cert Serial              : None
2025-08-06 07:58:49,722 - root - INFO - DMZ Gateway Cert Subject             : None
2025-08-06 07:58:49,722 - root - INFO - INTERNAL Gateway Cert                : None
2025-08-06 07:58:49,722 - root - INFO - INTERNAL Gateway Cert Issuer         : None
2025-08-06 07:58:49,722 - root - INFO - INTERNAL Gateway Cert Serial         : None
2025-08-06 07:58:49,722 - root - INFO - INTERNAL Gateway Cert Subject        : None
2025-08-06 07:58:49,722 - root - INFO - ****************************************************
2025-08-06 07:58:49,722 - root - INFO - **************  Database properties  ***************
2025-08-06 07:58:49,722 - root - INFO - ****************************************************
2025-08-06 07:58:49,722 - root - INFO - Create readonly DB connection : false
2025-08-06 07:58:49,722 - root - INFO - Create client read DB connection : false
2025-08-06 07:58:49,722 - root - INFO - Properties         : {'maximumPoolSize': 15, 'minimumPoolSize': 3}
2025-08-06 07:58:49,722 - root - INFO - Connection Properties         : na
2025-08-06 07:58:49,722 - root - INFO - User                          : admin
2025-08-06 07:58:49,722 - root - INFO - MySql JDBC Url                : jdbc:mysql://layer7-dev-test-instance-1.c70as4g0g08k.eu-west-3.rds.amazonaws.com:3306/otk_db
2025-08-06 07:58:49,722 - root - INFO - MySql JDBC driver             : com.mysql.jdbc.Driver
2025-08-06 07:58:49,723 - root - INFO - ****************************************************
2025-08-06 07:58:49,723 - root - INFO - ****************************************************
2025-08-06 07:58:49,723 - root - INFO - ****************************************************
2025-08-06 07:58:49,723 - root - INFO - Trying to connect to restman https://my-ssg-gateway:8443/restman/1.0/doc/home.html With retry count 10 and backoff factor 0.1
2025-08-06 07:58:50,753 - urllib3.connectionpool - WARNING - Retrying (Retry(total=9, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d714d050>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:58:51,979 - urllib3.connectionpool - WARNING - Retrying (Retry(total=8, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5c72210>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:58:53,398 - urllib3.connectionpool - WARNING - Retrying (Retry(total=7, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5c72b90>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:58:55,202 - urllib3.connectionpool - WARNING - Retrying (Retry(total=6, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5c734d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:58:57,830 - urllib3.connectionpool - WARNING - Retrying (Retry(total=5, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5c73e10>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:59:01,032 - urllib3.connectionpool - WARNING - Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5b947d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:59:08,454 - urllib3.connectionpool - WARNING - Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5b95150>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:59:22,274 - urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5b95ad0>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:59:48,898 - urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7ff2d5b96450>: Failed to establish a new connection: [Errno 111] Connection refused')': /restman/1.0/doc/home.html
2025-08-06 07:59:49,040 - root - INFO - Connected to restman
2025-08-06 07:59:49,040 - root - INFO - Checking if database connection exists with name OAuth
2025-08-06 07:59:49,111 - root - INFO - Can not find database connection with name: OAuth
2025-08-06 07:59:49,170 - root - INFO - Current installed Version : None
2025-08-06 07:59:49,170 - root - INFO - Generating bundle for otk connection
2025-08-06 07:59:49,293 - root - INFO - Created otk database connection with goid: 4432207d16a1b505e8a6ed59993eaa24 with status code 201
2025-08-06 07:59:49,294 - root - INFO - Installing solution kit sub module: ['OTK Assertions', 'OTK Configuration'] With maximum attempts 10 and backoff factor 0.1
2025-08-06 07:59:54,312 - root - INFO - Installed OTK solution kit sub module: ['OTK Assertions', 'OTK Configuration'] with status code 200
2025-08-06 07:59:54,314 - root - INFO - Installing solution kit sub module: ['Shared OAuth Resources'] With maximum attempts 10 and backoff factor 0.1
2025-08-06 08:01:14,014 - root - INFO - Installed OTK solution kit sub module: ['Shared OAuth Resources'] with status code 200
2025-08-06 08:01:14,016 - root - INFO - OTK solution kit sub module: Shared Portal Resources is not vaid for profile SINGLE with mysql database - tag portal
2025-08-06 08:01:14,016 - root - INFO - OTK solution kit sub module: Persistence Layer: Cassandra is not vaid for profile SINGLE with mysql database - tag cass
2025-08-06 08:01:14,016 - root - INFO - Installing solution kit sub module: ['Persistence Layer: MySQL or Oracle'] With maximum attempts 10 and backoff factor 0.1
2025-08-06 08:01:30,207 - root - INFO - Installed OTK solution kit sub module: ['Persistence Layer: MySQL or Oracle'] with status code 200
2025-08-06 08:01:30,208 - root - INFO - OTK solution kit sub module: Portal Persistence Layer: MySQL or Oracle is not vaid for profile SINGLE with mysql database - tag mysql_portal
2025-08-06 08:01:30,209 - root - INFO - OTK solution kit sub module: Portal Persistence Layer: Cassandra is not vaid for profile SINGLE with mysql database - tag cass_portal
2025-08-06 08:01:30,209 - root - INFO - Installing solution kit sub module: ['Internal: OAuth Validation Point', 'DMZ: OAuth 2.0 and OpenID Connect endpoints', 'Internal: Server Tools'] With maximum attempts 10 and backoff factor 0.1
2025-08-06 08:02:14,746 - root - INFO - Installed OTK solution kit sub module: ['Internal: OAuth Validation Point', 'DMZ: OAuth 2.0 and OpenID Connect endpoints', 'Internal: Server Tools'] with status code 200
2025-08-06 08:02:14,748 - root - INFO - OTK solution kit sub module: Internal: Portal is not vaid for profile SINGLE with mysql database - tag portal
2025-08-06 08:02:14,748 - root - INFO - Installing solution kit sub module: ['Internal: Endpoint to access the client persistence layer', 'Internal: Endpoint to access the session persistence layer', 'Internal: Endpoint to access the token persistence layer'] With maximum attempts 10 and backoff factor 0.1
2025-08-06 08:02:19,694 - root - INFO - Installed OTK solution kit sub module: ['Internal: Endpoint to access the client persistence layer', 'Internal: Endpoint to access the session persistence layer', 'Internal: Endpoint to access the token persistence layer'] with status code 200
2025-08-06 08:02:19,695 - root - INFO - Generating bundle for updating CWP
Traceback (most recent call last):
  File "/docker/./sk-upgrade/install_otk.py", line 900, in <module>
2025-08-06 08:02:19,756 - root - ERROR - Failed create or update otk.dbsystem 400 Response: b'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>\n<l7:Error xmlns:l7="http://ns.l7tech.com/2010/04/gateway-management">\n    <l7:Type>DuplicateObject</l7:Type>\n    <l7:TimeStamp>2025-08-06T08:02:19.753Z</l7:TimeStamp>\n    <l7:Link rel="self" uri="https://my-ssg-gateway-795fdb97f5-lg8zw:8443/restman/1.0/clusterProperties/0c3729ca9f24cb4f2f55d363c7f7c51a"/>\n    <l7:Detail>(name)  must be unique</l7:Detail>\n</l7:Error>\n'
    main()
  File "/docker/./sk-upgrade/install_otk.py", line 872, in main
    postInstallationTasks()
  File "/docker/./sk-upgrade/install_otk.py", line 646, in postInstallationTasks
    create_otk_db_system(OTK_DATABASE_TYPE)
  File "/docker/./sk-upgrade/install_otk.py", line 348, in create_otk_db_system
    validate_response(200, response , 'Failed create or update otk.dbsystem')
  File "/docker/./sk-upgrade/install_otk.py", line 161, in validate_response
    raise Exception(f'{error_message}')
Exception: Failed create or update otk.dbsystem

Using the policy manager, SSG and OTK seems ok:

Environment

Container API Gateway 

Resolution

The error occurs because the OTK installer is attempting to create a cluster-wide property (otk.dbsystem) that already exists, and the property name must be unique.

<l7:Type>DuplicateObject</l7:Type>
<l7:Detail>(name) must be unique</l7:Detail>
...
Exception: Failed create or update otk.dbsystem

- If you've already run the OTK install once (successfully or not), some objects such as cluster-wide properties (otk.dbsystem) may already exist in the API Gateway's configuration or database.
- When deploying to a fresh external database, but with lingering cluster properties from a previous install (possibly from a former run or attempt), this error can surface.
- By default, the installer doesn't overwrite existing solution kits or cluster properties unless you explicitly force it. (Not using forceInstallOrUpgrade)

- To avoid this issue, search for any existing otk.dbsystem property or OTK-related cluster properties and carefully delete them if they are leftovers from a prior installation attempt.
- If you intend to start fresh, set the forceInstallOrUpgrade value to true, which should instruct the installer to clean up and recreate existing OTK solution kit entities, including cluster properties.
Example:
otk:
  forceInstallOrUpgrade: true

Powered by Wolken Software