Application Key Sync is failing after Portal upgrade

Products

CA API Developer Portal

Issue/Introduction

Application Key Sync is failing after Portal upgrade from 5.2.3 to 5.3 .

{"package":"com.l7tech.external.assertions.portaldeployer.server.client.MessageProcessor","level":"INFO","log":{"message":"processing message with messageId(traceId)\u003d6109922103f920c7fcaa37ae783f823d"},"time":"2025-12-01T16:25:04.017+0000"}

{"package":"com.l7tech.server.jdbc.JdbcQueryingManagerImpl","level":"WARNING","log":{"service":"Portal API Key Storage Service [/portalman/api-keys/storage*]","user-id":"0000000000000000fffffffffffffffe:0cc1e2ecfc467ae8ed9832043b017611","client-ip":"127.0.0.1","request-id":"df317c77e63161fd-13d7848f184b8865","service-folder-path":"/API Portal Integration","message":"Failed to perform querying since Duplicate entry l7d3a4ed36093b470c9684664b4fab56ed for key oauth_client_key.PRIMARY","listen-port":"Default HTTPS (8443)"},"time":"2025-12-01T16:25:04.333+0000","otelId":"35f5937cab0018f6e261df3a34950a49-6f4aaadef9cd7d1e"}

Environment

API Developer Portal 5.3 (Kubernetes)

Cause

After Portal DB upgrade, some apikeys are missing which still exist in OTK, the customer tried to re-create the apikey in portal, the apikeys are created but with a new apikey_pk, which caused the apikey deployment failed. The apikey deployment requires that if the apikey already exists in OTK, both apikey and apikey_pk must match between Portal and OTK.

Resolution

Delete duplicate apikeys from OTK and resync with Portal.

Steps to delete duplicate apikey:

ApiKey: l7d3a4ed36093b470c9684664b4fab56ed as example:

Verify organization is acceptable between GW and Portal:
1. login OTK on GW side and run mysql> SELECT * FROM portal_apikey where apikey = 'l7d3a4ed36093b470c9684664b4fab56ed' \G;
  keep track the apikey_pk value as <apikey_pk_1>
2. open https://portaldomain.com/publish/api-proxy/details/af48c6b1-7cca-463d-9f8d-1f4cdba64cf0/keys?status=ERROR and click same apikey name, which bring you to the application page.
3. compare organization name is matched between step a and step b, (if not matched still OK, as long as the the discrepancy is acceptable by the customer)
Backup data from OTK before performing any changes.

mysqldump -t -u <user> -p<password> <otk db> portal_apikey --complete-insert --where="apikey_pk in ('<apikey_pk_1>')" > portal_apikey_insert_<apikey_pk_1>.sql
mysqldump -t -u <user> -p<password> <otk db> oauth_client_key --complete-insert --where="client_ident in ('<apikey_pk_1>')" > oauth_client_key_insert_<apikey_pk_1>.sql
mysqldump -t -u <user> -p<password> <otk db> oauth_client --complete-insert --where="client_ident in ('<apikey_pk_1>')" > oauth_client_insert_<apikey_pk_1>.sql
Remove data from OTK
1. in OTK db run: mysql> SELECT * FROM otk.oauth_client_key where client_ident = '<apikey_pk_1>'\G and keep track of client_name
2. login https://<gateway hostname>:<port>/oauth/manager
3. search client_name and to find out the apikey 'l7d3a4ed36093b470c9684664b4fab56ed' and delete it.
4. remove the apikey from OTK.portal_apikey: mysql> delete from portal_apikey where apikey_pk in ('<apikey_pk_1>');
Run mysql> SELECT * FROM portal_apikey where apikey = 'l7d3a4ed36093b470c9684664b4fab56ed' \G; again you will see a new apikey_pk
a. with the new value you can see more detail api key information in portal db: mysql> SELECT * FROM portal.APPLICATION_API_KEY where UUID = '<new apikey_pk>' \G;

repeat above steps from the list in page https://portaldomain.com/publish/api-proxy/details/af48c6b1-7cca-463d-9f8d-1f4cdba64cf0/keys?status=ERROR

If there are more API keys with deployment errors, copy and paste all duplicate API keys—which can be queried from the OTK database—to obtain all the apikey_pk values. Then take a full backup of the OTK database, delete all entries from OTK.portal_apikey, and save the redeployment step for the end. Finally, trigger a bulk deployment all at once by switching the proxy key deployment type from On-Demand to Automatic.