Upgrade "gateway" failed: timed out waiting for the condition

Products

CA API Gateway

Issue/Introduction

During the upgrade of API Gateway deployment from v11.0.00_CR2 to v11.2.1 in AWS EKS environment.

The deployment pipeline reports a failure and times out while waiting for the upgrade to complete

Seems to be hung state and we couldn't re-deploy charts again.

2026-07-02T23:00:59.217Z] Writing manifest to image destination
[2026-07-02T23:01:30.086Z] Added new context arn:aws:eks:us-east-1:713927983851:cluster/ciam-wam-c2-us-e1 to /mnt/apps/.kube/config
[2026-07-02T23:01:30.591Z] REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
[2026-07-02T23:01:30.591Z] 79 Thu May 7 20:28:01 2026 failed gateway-3.0.26 11.0.00_CR2 Upgrade "gateway" failed: timed out waiting for the condition
[2026-07-02T23:01:30.591Z] 80 Thu May 7 20:38:03 2026 superseded gateway-3.0.26 11.0.00_CR2 Rollback to 78
[2026-07-02T23:01:30.591Z] 81 Thu May 7 20:47:26 2026 failed gateway-3.0.26 11.0.00_CR2 Upgrade "gateway" failed: timed out waiting for the condition
[2026-07-02T23:01:30.591Z] 82 Thu May 7 20:57:28 2026 superseded gateway-3.0.26 11.0.00_CR2 Rollback to 80
[2026-07-02T23:01:30.591Z] 83 Thu May 7 21:11:51 2026 superseded gateway-3.0.26 11.0.00_CR2 Upgrade complete
[2026-07-02T23:01:30.591Z] 84 Thu Jul 2 20:33:42 2026 failed gateway-3.1.1 11.2.1 Upgrade "gateway" failed: timed out waiting for the condition
[2026-07-02T23:01:30.591Z] 85 Thu Jul 2 20:43:44 2026 deployed gateway-3.0.26 11.0.00_CR2 Rollback to 83
[2026-07-02T23:01:30.591Z] 86 Thu Jul 2 20:57:10 2026 failed gateway-3.1.1 11.2.1 Upgrade "gateway" failed: timed out waiting for the condition
[2026-07-02T23:01:30.591Z] 87 Thu Jul 2 21:07:12 2026 failed gateway-3.0.26 11.0.00_CR2 Release "gateway" failed: timed out waiting for the condition
[2026-07-02T23:01:30.591Z] 88 Thu Jul 2 21:18:49 2026 pending-upgrade gateway-3.1.1 11.2.1 Preparing upgrade
[2026-07-02T23:01:30.591Z] ==> Linting gateway
[2026-07-02T23:01:30.591Z] Error unable to check Chart.yaml file in chart: stat gateway/Chart.yaml: no such file or directory
[2026-07-02T23:01:30.591Z]
[2026-07-02T23:01:30.591Z] Error: 1 chart(s) linted, 1 chart(s) failed
[2026-07-02T23:01:31.652Z] history.go:56: [debug] getting history for release gateway
[2026-07-02T23:01:31.652Z] upgrade.go:123: [debug] preparing upgrade for gateway
[2026-07-02T23:01:31.652Z] Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
[2026-07-02T23:01:31.652Z] helm.go:81: [debug] another operation (install/upgrade/rollback) is in progress
[2026-07-02T23:01:31.652Z] helm.sh/helm/v3/pkg/action.init
[2026-07-02T23:01:31.652Z] helm.sh/helm/v3/pkg/action/action.go:62
[2026-07-02T23:01:31.652Z] runtime.doInit
[2026-07-02T23:01:31.652Z] runtime/proc.go:6265
[2026-07-02T23:01:31.652Z] runtime.doInit
[2026-07-02T23:01:31.652Z] runtime/proc.go:6242
[2026-07-02T23:01:31.652Z] runtime.main
[2026-07-02T23:01:31.652Z] runtime/proc.go:208
[2026-07-02T23:01:31.652Z] runtime.goexit
[2026-07-02T23:01:31.652Z] runtime/asm_amd64.s:1371
[2026-07-02T23:01:31.652Z] UPGRADE FAILED

error observed:

m8s        Warning   FailedCreate             replicaset/gateway-7editedc98                         Error creating: pods "gateway-7edited7c98-2rv8c" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=8, used: limits.cpu=64300m, limited: limits.cpu=70
2m8s        Warning   FailedCreate replicaset/gateway-7edited7c98                         Error creating: pods "gateway-7edited7c98-2mbfx" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=8, used: limits.cpu=64300m, limited: limits.cpu=70
2m8s        Warning   FailedCreate replicaset/gateway-7edited7c98                         Error creating: pods "gateway-7editedc98-7rk6k" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=8, used: limits.cpu=64300m, limited: limits.cpu=70

Environment

CA API Gateway v11.0.00_CR2

CA API Gateway 11.2

Platform: AWS EKS

Previous Version: gateway-3.0.26 / API Gateway 11.0.00_CR2

Target Version: gateway-3.1.1 / API Gateway 11.2.1

Cause

Revision 88 is stuck in pending-upgrade status in Helm's release store (Kubernetes secret sh.helm.release.v1.gateway.v88).
Helm locks the release during any operation — because revision 88 never transitioned out of pending-upgrade,
all subsequent operations fail with "another operation is in progress."

Resolution

1. Confirm the stuck secret (read-only, safe to run)

kubectl get secret -n <gateway-namespace> | grep sh.helm.release.v1.gateway

You'll see a secret for revision v88 with status pending-upgrade.

2. Delete the stuck pending-upgrade secret

kubectl delete secret sh.helm.release.v1.gateway.v88 -n <gateway-namespace>

This tells Helm the lock is gone. It does NOT affect running pods — it only removes the stuck state record.

3. Verify the release history is now clean

helm history gateway -n <gateway-namespace>

Revision 88 should be gone. Revision 85 (deployed, gateway-3.0.26) should now be the current state.

4. Rollback to the last known-good revision explicitly :

example rollback to revision 85.

helm rollback gateway 85 -n <gateway-namespace> --wait

This re-establishes a clean deployed state before retrying the upgrade.

5. Then retry the upgrade to 11.2.1

helm upgrade gateway gateway-3.1.1 -n <gateway-namespace> --wait --timeout 15m