How to resolve NGINX Connection Exhaustion with the error "worker_connections are not enough" for ArgoCD managing many vSphere Namespaces
search cancel

How to resolve NGINX Connection Exhaustion with the error "worker_connections are not enough" for ArgoCD managing many vSphere Namespaces

book

Article ID: 449969

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

ArgoCD related operations get failure, such as syncing the ArgoCD application. Snippet from ArgoCD application Pod logs is as below. 

:
time="2025-06-08T05:49:54Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference\n
goroutine 203 [running]:\n
runtime/debug.Stack()\n
\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\n
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()\n
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1605 +0x54\n
panic({0x5640a0af52e0?, 0x5640a3c3e730?})\n
\t/usr/local/go/src/runtime/panic.go:791 +0x132\n
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000d7c960, 0xc004b1cc08, 0xc000987d48, {0xc00569b810, 0x1, 0x1}, {0xc003c17d80, 0x1, 0x1}, 0x0, ...)\n
\t/go/src/github.com/argoproj/argo-cd/controller/state.go:952 +0x6019\n
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0008501e0)\n
:
:
\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161\n
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 98\n
\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:899 +0x89b\n


If directly using the "kubectl get <resource>" from the supervisor cluster, it shows following error message.

# kubectl get pods -A 
Get "https://yy.yy.y.yy:443/api/v1/pods?limit=500": read tcp xx.xx.xx.xx:33286->yy.yy.yy.yy:443: read: connection reset by peer - error from a previous attempt: read tcp xx.xx.xx.xx:33284->yy.yy.yy.yy:443: read: connection reset by peer

Environment

VMware vSphere Kubernetes Service
ArgoCD Supervisor Service 1.0.0, 1.0.1 and 1.1.0

Cause

Checking the logs of kubectl-plugin-vsphere pod under kube-system namespace, there are many alerts of "worker_connections are not enough".

Below log snippet was from before increasing the nginx connection number to 1024. (Default was 512)
In the current Supervisor Cluster, the number of nginx connection is increased to 1024.

YYYY/MM/DD 06:32:33 [alert] 7#0: *248771 512 worker_connections are not enough while connecting to upstream, client: xx.xx.xx.xx, server: default, request: "GET /apis/vmoperator.vmware.com/v1alpha3/namespaces/argocd10/virtualmachineservices?limit=500 HTTP/2.0", upstream: "https://127.0.0.1:6443/apis/vmoperator.vmware.com/v1alpha3/namespaces/argocd10/virtualmachineservices?limit=500", host: "yy.yy.yy.yy:443" 
YYYY/MM/DD 06:32:33 [alert] 7#0: 512 worker_connections are not enough
YYYY/MM/DD 06:32:34 [alert] 7#0: *248723 512 worker_connections are not enough while connecting to upstream, client: xx.xx.xx.xx, server: default, request: "GET /apis/mobility-operator.vmware.com/v1alpha1/namespaces/argocd11/importoperations?limit=500 HTTP/2.0", upstream: "https://127.0.0.1:6443/apis/mobility-operator.vmware.com/v1alpha1/namespaces/argocd11/importoperations?limit=500", host: "yy.yy.yy.yy:443" zz.zz.zz.zz - - [09/Jun/2025:06:32:34 +0000] "GET /apis/mobility-operator.vmware.com/v1alpha1/namespaces/argocd11/importoperations?limit=500 HTTP/2.0" 500 177 "-" "argocd-application-controller/v0.0.0 (linux/amd64) kubernetes/$Format"

The root cause of this issue is that ArgoCD itself setup multi-watch connections with Supervisor Cluster with its VIP on each managed vSphere namespace. When ArgoCD pod communicates with Supervisor Cluster with its VIP, it goes through the nginx, which only supports HTTP/1.1.  The number of ArgoCD watch connections occupies all the 1024 worker connections.

Resolution

Below steps could be too difficult for some users to follow, in that case, please open a case with Broadcom support and mention this KB. 

There are two options to fix the issue as below:

Option 1: Customize Inclusion List in ArgoCD Spec - Recommended
    Pros: Quick mitigation without altering network paths or Carvel package overlay configurations.
    Cons: Reduces watch traffic significantly - down to ~14 watch connections per vSphereNamespace, if customers just want to use ArgoCD to deploy VKS into the vSphere namespace customers can define an explicit resource inclusion list directly inside the ArgoCD Custom Resource (CR) spec. By restricting ArgoCD to watch only the resource types explicitly required (for instance, when using ArgoCD solely to deploy VKS into a vSphereNamespace), ArgoCD drops unneeded watches for other vSphere CRDs.
    Detail steps: 

- The ArgoCD instance manifest to include VKS Cluster, Addon:

apiVersion: argocd-service.vsphere.vmware.com/v1alpha1
kind: ArgoCD
metadata:
  name: argocd-sample
  namespace: argocd-ns
spec:
  version: 
  resourceConfig:
    inclusions:
    - apiGroups:
      - addons.kubernetes.vmware.com
      kinds:
      - AddonConfig 
      - AddonInstall 
      - ClusterAddon
      clusters:
      - https://###.###.###.###:443
    - apiGroups:
      - cert-manager.io
      kinds:
      - Certificate
      - Issuer
      clusters:
      - https://###.###.###.###:443
    - apiGroups:
      - cluster.x-k8s.io
      kinds:
      - ClusterClass
      - Cluster
      - MachineDeployment
      - MachineHealthCheck
      - Machine
      - MachineSet
      clusters:
      - https://###.###.###.###:443
    - apiGroups:
      - controlplane.cluster.x-k8s.io
      kinds:
      - KubeadmControlPlane
      clusters:
      - https://###.###.###.###:443
    - apiGroups:
      - cni.tanzu.vmware.com   ==> this could be removed, if customers do not directly save its manifest in the Git Repo
      kinds:
      - AntreaConfig
      - CalicoConfig
      clusters:
      - https://###.###.###.###:443

the clusters field is optional:

  • If you omit the clusters field (or leave it empty), the inclusion/exclusion rule will apply to all clusters managed by Argo CD.
  • If you specify a cluster URL, the rule will only apply to that specific cluster.
  • the cluster's URL is retrived from argocd cluster list or directly from argocd UI.
➜  ~ argocd cluster list
SERVER                                      NAME             VERSION  STATUS      MESSAGE  PROJECT
https://###.###.###.###:443 (3 namespaces)  ###.###.###.###  1.32     Successful

 

- Using this argocd instance to create applications for VKS cluster, Addon 

[YYYY-MM-DDT08:47:53.836Z] INFO:root:====== 218  CMD: argocd app create vks-cluster-e2e --repo git@<your-github-url>:vcf/vkd-argocd-test.git --path vks-cluster-e2e --dest-server https://###.###.###.###:443 --dest-namespace argocd-e2e-vks-cluster --revision topic/argoe2e/vks-cluster-1234567890
[YYYY-MM-DDT08:47:55.732Z] application 'vks-cluster-e2e' unchanged
[YYYY-MM-DDT08:47:55.732Z] INFO:root:====== 219  CMD: argocd app get vks-cluster-e2e
[YYYY-MM-DDT08:47:56.299Z] Name:               argocd-e2e-vks-cluster/vks-cluster-e2e
[YYYY-MM-DDT08:47:56.299Z] Project:            default
[YYYY-MM-DDT08:47:56.299Z] Server:             https://###.###.###.###:443
[YYYY-MM-DDT08:47:56.299Z] Namespace:          argocd-e2e-vks-cluster
[YYYY-MM-DDT08:47:56.299Z] URL:                https://###.###.###.###/applications/vks-cluster-e2e
[YYYY-MM-DDT08:47:56.299Z] Source:
[YYYY-MM-DDT08:47:56.299Z] - Repo:             git@<your-github-url>:vcf/vkd-argocd-test.git
[YYYY-MM-DDT08:47:56.299Z]   Target:           topic/argoe2e/vks-cluster-1234567890
[YYYY-MM-DDT08:47:56.299Z]   Path:             vks-cluster-e2e
[YYYY-MM-DDT08:47:56.299Z] SyncWindow:         Sync Allowed
[YYYY-MM-DDT08:47:56.299Z] Sync Policy:        Manual
[YYYY-MM-DDT08:47:56.299Z] Sync Status:        Synced to topic/argoe2e/vks-cluster-1234567890 (#######)
[YYYY-MM-DDT08:47:56.299Z] Health Status:      Healthy
[YYYY-MM-DDT08:47:56.299Z] 
[YYYY-MM-DDT08:47:56.299Z] GROUP             KIND     NAMESPACE               NAME             STATUS  HEALTH   HOOK  MESSAGE
[YYYY-MM-DDT08:47:56.299Z] cluster.x-k8s.io  Cluster  argocd-e2e-vks-cluster  vks-cluster-e2e  Synced  Healthy        cluster.cluster.x-k8s.io/vks-cluster-e2e created
[YYYY-MM-DDT08:47:56.299Z] INFO:root:====== 220  CMD: argocd app sync vks-cluster-e2e 
[YYYY-MM-DDT08:47:57.673Z] TIMESTAMP                  GROUP                   KIND   NAMESPACE                              NAME    STATUS   HEALTH        HOOK  MESSAGE
[YYYY-MM-DDT08:47:57.673Z] YYYY-MM-DDT08:47:56+00:00  cluster.x-k8s.io     Cluster  argocd-e2e-vks-cluster       vks-cluster-e2e    Synced  Healthy             
[YYYY-MM-DDT08:48:01.852Z] YYYY-MM-DDT08:48:00+00:00  cluster.x-k8s.io     Cluster  argocd-e2e-vks-cluster       vks-cluster-e2e    Synced  Healthy              cluster.cluster.x-k8s.io/vks-cluster-e2e configured
[YYYY-MM-DDT08:48:02.416Z] YYYY-MM-DDT08:48:01+00:00  addons.kubernetes.vmware.com  AddonConfig  argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  OutOfSync                       
[YYYY-MM-DDT08:48:03.829Z] YYYY-MM-DDT08:48:03+00:00  cluster.x-k8s.io                 Cluster   argocd-e2e-vks-cluster       vks-cluster-e2e            Synced   Healthy              Cluster is Running
[YYYY-MM-DDT08:48:03.829Z] YYYY-MM-DDT08:48:03+00:00  addons.kubernetes.vmware.com  AddonConfig  argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  OutOfSync                       addonconfig.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:04.398Z] YYYY-MM-DDT08:48:03+00:00  addons.kubernetes.vmware.com  AddonInstall  argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager   Running   Synced              addoninstall.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:04.398Z] YYYY-MM-DDT08:48:03+00:00  addons.kubernetes.vmware.com  AddonInstall  argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  OutOfSync  Unknown              addoninstall.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:06.923Z] 
[YYYY-MM-DDT08:48:06.923Z] Name:               argocd-e2e-vks-cluster/vks-cluster-e2e
[YYYY-MM-DDT08:48:06.923Z] Project:            default
[YYYY-MM-DDT08:48:06.923Z] Server:             https://###.###.###.###:443
[YYYY-MM-DDT08:48:06.923Z] Namespace:          argocd-e2e-vks-cluster
[YYYY-MM-DDT08:48:06.923Z] URL:                https://###.###.###.###/applications/vks-cluster-e2e
[YYYY-MM-DDT08:48:06.923Z] Source:
[YYYY-MM-DDT08:48:06.923Z] - Repo:             git@<your-github-url>:vcf/vkd-argocd-test.git
[YYYY-MM-DDT08:48:06.923Z]   Target:           topic/argoe2e/vks-cluster-1234567890
[YYYY-MM-DDT08:48:06.923Z]   Path:             vks-cluster-e2e
[YYYY-MM-DDT08:48:06.923Z] SyncWindow:         Sync Allowed
[YYYY-MM-DDT08:48:06.923Z] Sync Policy:        Manual
[YYYY-MM-DDT08:48:06.923Z] Sync Status:        Synced to topic/argoe2e/vks-cluster-1234567890 (#######)
[YYYY-MM-DDT08:48:06.923Z] Health Status:      Healthy
[YYYY-MM-DDT08:48:06.923Z] 
[YYYY-MM-DDT08:48:06.923Z] Operation:          Sync
[YYYY-MM-DDT08:48:06.923Z] Sync Revision:      #######9683262ad4697babb7c17dde35efcd8d1
[YYYY-MM-DDT08:48:06.923Z] Phase:              Succeeded
[YYYY-MM-DDT08:48:06.923Z] Start:              YYYY-MM-DD 08:47:56 +0000 UTC
[YYYY-MM-DDT08:48:06.923Z] Finished:           YYYY-MM-DD 08:48:04 +0000 UTC
[YYYY-MM-DDT08:48:06.923Z] Duration:           8s
[YYYY-MM-DDT08:48:06.923Z] Message:            successfully synced (all tasks run)
[YYYY-MM-DDT08:48:06.923Z] 
[YYYY-MM-DDT08:48:06.923Z] GROUP                         KIND          NAMESPACE               NAME                          STATUS  HEALTH   HOOK  MESSAGE
[YYYY-MM-DDT08:48:06.923Z] cluster.x-k8s.io              Cluster       argocd-e2e-vks-cluster  vks-cluster-e2e               Synced  Healthy        Cluster is Running
[YYYY-MM-DDT08:48:06.923Z] addons.kubernetes.vmware.com  AddonConfig   argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  Synced                 addonconfig.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:06.923Z] addons.kubernetes.vmware.com  AddonInstall  argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  Synced  Healthy        addoninstall.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:10.205Z] INFO:root:====== 221  CMD: argocd app wait vks-cluster-e2e
[YYYY-MM-DDT08:48:10.463Z] 
[YYYY-MM-DDT08:48:10.463Z] Name:               argocd-e2e-vks-cluster/vks-cluster-e2e
[YYYY-MM-DDT08:48:10.463Z] Project:            default
[YYYY-MM-DDT08:48:10.463Z] Server:             https://###.###.###.###:443
[YYYY-MM-DDT08:48:10.463Z] Namespace:          argocd-e2e-vks-cluster
[YYYY-MM-DDT08:48:10.463Z] URL:                https://###.###.###.###/applications/vks-cluster-e2e
[YYYY-MM-DDT08:48:10.463Z] Source:
[YYYY-MM-DDT08:48:10.463Z] - Repo:             git@<your-github-url>:vcf/vkd-argocd-test.git
[YYYY-MM-DDT08:48:10.463Z]   Target:           topic/argoe2e/vks-cluster-1234567890
[YYYY-MM-DDT08:48:10.463Z]   Path:             vks-cluster-e2e
[YYYY-MM-DDT08:48:10.463Z] SyncWindow:         Sync Allowed
[YYYY-MM-DDT08:48:10.463Z] Sync Policy:        Manual
[YYYY-MM-DDT08:48:10.463Z] Sync Status:        Synced to topic/argoe2e/vks-cluster-1234567890 (#######)
[YYYY-MM-DDT08:48:10.463Z] Health Status:      Healthy
[YYYY-MM-DDT08:48:10.463Z] 
[YYYY-MM-DDT08:48:10.463Z] Operation:          Sync
[YYYY-MM-DDT08:48:10.463Z] Sync Revision:      #######9683262ad4697babb7c17dde35efcd8d1
[YYYY-MM-DDT08:48:10.463Z] Phase:              Succeeded
[YYYY-MM-DDT08:48:10.463Z] Start:              YYYY-MM-DD 08:47:56 +0000 UTC
[YYYY-MM-DDT08:48:10.463Z] Finished:           YYYY-MM-DD 08:48:04 +0000 UTC
[YYYY-MM-DDT08:48:10.463Z] Duration:           8s
[YYYY-MM-DDT08:48:10.463Z] Message:            successfully synced (all tasks run)
[YYYY-MM-DDT08:48:10.463Z] 
[YYYY-MM-DDT08:48:10.463Z] GROUP                         KIND         NAMESPACE               NAME                          STATUS  HEALTH   HOOK  MESSAGE
[YYYY-MM-DDT08:48:10.463Z] cluster.x-k8s.io              Cluster      argocd-e2e-vks-cluster  vks-cluster-e2e               Synced  Healthy        Cluster is Running
[YYYY-MM-DDT08:48:10.463Z] addons.kubernetes.vmware.com  AddonConfig  argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  Synced                 addonconfig.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:10.463Z] addons.kubernetes.vmware.com  AddonInstall argocd-e2e-vks-cluster  vks-cluster-e2e-cert-manager  Synced  Healthy        addoninstall.addons.kubernetes.vmware.com/vks-cluster-e2e-cert-manager created
[YYYY-MM-DDT08:48:11.048Z] INFO:root:App sync status: Synced, health status: Healthy
    


Option 2: Apply a ytt Overlay for In-Cluster Management
    Pros: Environments where ArgoCD should bypass external NGINX proxies completely by talking to the local Kubernetes API server endpoint directly. Using a Carvel ytt overlay reconfigures ArgoCD to communicate with the local API server endpoint (https://kubernetes.default.svc) rather than routing traffic through external vSphere NGINX Ingress proxies.
    Cons: The configuration is complex.
    Detail steps: 

Due to the nginx server has 1024 hard limit connections, we could workaround of using in-cluster service endpoint (https://kubernetes.default.svc).

  1. Set inCluster as true in the argocd-cm (this is only for argocd-service 1.1.0)
  2. Create the secret in to the argocd service namespace, similar as svc-argocd-service-####
  3. cat << EOF | kubectl apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: incluster-overlay
      namespace: svc-argocd-service-#####
    stringData:
      patch.yaml: |
        #@ load("@ytt:overlay", "overlay")
        #@overlay/match by=overlay.subset({"kind": "ConfigMap", "metadata": {"name": "argocd-cm"}})
        ---
        data:
          #@overlay/match missing_ok=True
          cluster.inClusterEnabled: "true"
    EOF
  4. Then get the pkgi and patch the pkgi with the following command
  5. root@<Supervisor> # kubectl get pkgi -n svc-argocd-service-#####
    NAME                   PACKAGE NAME                   PACKAGE VERSION         DESCRIPTION           AGE     PAUSED
    ns-1---argocd-1-pkgi   argocd.kubernetes.vmware.com   3.0.19+vmware.1-vks.1   Reconcile succeeded   4m44s

    root@<Supervisor> # kubectl annotate pkgi ns-1---argocd-1-pkgi ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=incluster-overlay -n svc-argocd-service-#####
    packageinstall.packaging.carvel.dev/ns-1---argocd-1-pkgi annotated
  6. Check the configmap data
  7. root@<Supervisor> # kubectl get cm argocd-cm -n ns-1 -o jsonpath='{.data}' |  jq '."cluster.inClusterEnabled"'
    "true"

  8. Now "argocd cluster list" command also shows the info

  9. ➜ vcfa argocd cluster list
    SERVER                                      NAME             VERSION  STATUS      MESSAGE                                                  PROJECT
    https://###.###.###.###:443 (2 namespaces)  ###.###.###.###  1.32     Successful
    https://kubernetes.default.svc              in-cluster                Unknown     Cluster has no applications and is not being monitored.
  10. Then run the same "argocd cluster add" command as in the doc Manage Resources in vSphere Namespaces with Argo CD
    Now you just need to add --in-cluster and --name in-cluster in the command line. 
  11. ~ argocd cluster add ###.###.###.### --namespace test-1 --namespace test-2 --in-cluster --name in-cluster --cluster-resources
    WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `###.###.###.###` with full namespace level privileges. Do you want to continue [y/N]? y
    supervisor cluster
    --system-namespace is not supported when it is Supervisor Cluster, we will create serviceaccount argocd-manager in ArgoCD namespace argocd-1 for kubernetes resource management
    Start permission checking for managing namespace of Supervisor Cluster
    {"level":"info","msg":"ServiceAccount \"argocd-manager\" already exists in namespace \"argocd-1\"","time":"YYYY-MM-DD:hh:mm:ss"}
    {"level":"info","msg":"RoleBinding \"test-1/argocd-1-argocd-manager-role-binding\" updated","time":"YYYY-MM-DD:hh:mm:ss"}
    {"level":"info","msg":"RoleBinding \"test-2/argocd-1-argocd-manager-role-binding\" updated","time":"YYYY-MM-DD:hh:mm:ss"}
    Namespace test-1, test-2 from Cluster 'https://kubernetes.default.svc' added
  12. "argocd cluster list"  show the info 
  13. ➜ vcfa argocd cluster list
    SERVER                                         NAME            VERSION  STATUS      MESSAGE                                                  PROJECT
    https://###.###.###.###:443 (2 namespaces)     ###.###.###.### 1.32     Successful
    https://kubernetes.default.svc (2 namespaces)  in-cluster               Unknown     Cluster has no applications and is not being monitored.
  14. Next update the existing argocd application to change the destination.name or destination.server to this in-cluster, as below.  
    User could also directly update it in the Argocd UI.
  15. ➜  vcfa kubectl get application -n argocd-1 guestbook-app -o yaml
    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
    ....
    spec:
      destination:
        namespace: test-2
        server: https://###.###.###.###:443 ========> change to https://kubernetes.default.svc
        (name: ###.###.###.### =======> in-cluster)
      project: default
      source:
        path: guestbook
        repoURL: https://github.com/<githubID>/magicblocks.git
        targetRevision: HEAD
  16. Then trigger the sync of this application.

  17.  In the ArgoCD UI,  you will see those applications changed to use in-cluster, instead of LB of Supervsior Clsuter

 

 

 

 

Powered by Wolken Software