How to resolve NGINX Connection Exhaustion for ArgoCD managing many vSphere Namespaces
Article ID: 449969
Updated On:
Products
Issue/Introduction
ArgoCD related operations get failure, such as syncing the ArgoCD application. Snippet from ArgoCD application Pod logs is as below.
:
time="2025-06-08T05:49:54Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference\n
goroutine 203 [running]:\n
runtime/debug.Stack()\n
\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\n
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()\n
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1605 +0x54\n
panic({0x5640a0af52e0?, 0x5640a3c3e730?})\n
\t/usr/local/go/src/runtime/panic.go:791 +0x132\n
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000d7c960, 0xc004b1cc08, 0xc000987d48, {0xc00569b810, 0x1, 0x1}, {0xc003c17d80, 0x1, 0x1}, 0x0, ...)\n
\t/go/src/github.com/argoproj/argo-cd/controller/state.go:952 +0x6019\n
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0008501e0)\n
:
:
\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161\n
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 98\n
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:899 +0x89b\n
If directly using the "kubectl get <resource>" from the supervisor cluster, it shows following error message.
# kubectl get pods -A
Get "https://yy.yy.y.yy:443/api/v1/pods?limit=500": read tcp xx.xx.xx.xx:33286->yy.yy.yy.yy:443: read: connection reset by peer - error from a previous attempt: read tcp xx.xx.xx.xx:33284->yy.yy.yy.yy:443: read: connection reset by peer
Environment
VMware vSphere Kubernetes Service
ArgoCD Supervisor Service 1.0.0, 1.0.1 and 1.1.0
Cause
Checking the logs of kubectl-plugin-vsphere pod under kube-system namespace, there are many alerts of "worker_connections are not enough".
Below log snippet was from before increasing the nginx connection number to 1024. (Default was 512)
In the current Supervisor Cluster, the number of nginx connection is increased to 1024.
2025/06/09 06:32:33 [alert] 7#0: *248771 512 worker_connections are not enough while connecting to upstream, client: xx.xx.xx.xx, server: default, request: "GET /apis/vmoperator.vmware.com/v1alpha3/namespaces/argocd10/virtualmachineservices?limit=500 HTTP/2.0", upstream: "https://127.0.0.1:6443/apis/vmoperator.vmware.com/v1alpha3/namespaces/argocd10/virtualmachineservices?limit=500", host: "yy.yy.yy.yy:443"
2025/06/09 06:32:33 [alert] 7#0: 512 worker_connections are not enough
2025/06/09 06:32:34 [alert] 7#0: *248723 512 worker_connections are not enough while connecting to upstream, client: xx.xx.xx.xx, server: default, request: "GET /apis/mobility-operator.vmware.com/v1alpha1/namespaces/argocd11/importoperations?limit=500 HTTP/2.0", upstream: "https://127.0.0.1:6443/apis/mobility-operator.vmware.com/v1alpha1/namespaces/argocd11/importoperations?limit=500", host: "yy.yy.yy.yy:443" zz.zz.zz.zz - - [09/Jun/2025:06:32:34 +0000] "GET /apis/mobility-operator.vmware.com/v1alpha1/namespaces/argocd11/importoperations?limit=500 HTTP/2.0" 500 177 "-" "argocd-application-controller/v0.0.0 (linux/amd64) kubernetes/$Format"
The root cause of this issue is that ArgoCD itself setup multi-watch connections with Supervisor Cluster with its VIP on each managed vSphere namespace. When ArgoCD pod communicates with Supervisor Cluster with its VIP, it goes through the nginx, which only supports HTTP/1.1. The number of ArgoCD watch connections occupies all the 1024 worker connections.
Resolution
Please open a case with Broadcom support and mention this KB.
Feedback
