No resources deployed in argocd instance namespace "User "system:serviceaccount:svc-argocd-service-domain-c10:argocd-service-controller-manager" cannot list resource "secrets" in API group "" at the cluster scope: service accounts in non-system namespaces

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

ArgoCD is deployed as a Supervisor Service. After applying the ArgoCD instance YAML in the Supervisor context, no Kubernetes resources will be visible under it in the Supervisor cluster.

Environment

VCF 9.0

Cause

• While deploying the ArgoCD Supervisor Service, if the “--cosign-signatures" flag is not included in the imgpkg copy command while moving the required images to the private repo, the required image signature verification will not be included as part of the overall ArgoCD service deployment.
• In such cases, the service may appear to be deployed successfully, and all pods in the operator namespace may be in a running state. However, no resources will be created in the ArgoCD namespace.
  
  $ kubectl get all -n argocd
No resources found in argocd namespace.
  
  
• You can observe related errors in the argocd-service-controller-manager pod logs, indicating the failure due to the missing permissions.
  
  

W0212 06:15:54.083708       1 reflector.go:569] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:svc-argocd-service-domain-c10:argocd-service-controller-manager" cannot list resource "secrets" in API group "" at the cluster scope: service accounts in non-system namespaces may not read secrets in all namespaces
E0212 06:15:54.083840       1 reflector.go:166] "Unhandled Error" err="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:251: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:svc-argocd-service-domain-c10:argocd-service-controller-manager\" cannot list resource \"secrets\" in API group \"\" at the cluster scope: service accounts in non-system namespaces may not read secrets in all namespaces" logger="UnhandledError"

Resolution

 

• Before deploying ArgoCD as a Supervisor Service, follow the documentation to copy all required images correctly to the private repository.

• Ensure that cosign signatures flag is included in the command. The signatures are mandatory for the ArgoCD service deployment.

          imgpkg copy -b ${TKGS_REPO_PATH} --to-tar tkg-service-v3.1.0.tar --cosign-signatures

• After uploading the image to your private repo, proceed with installing the ArgoCD Supervisor Service as per the documented steps.

• During installation, a pop-up displaying the image DigiCert should appear under compatibility messages

  • If the DigiCert is not displayed, it indicates that the cosign signature is missing.
    
    

• Even if the service appears to be deployed successfully and the operator namespace pods are running, no resources will be created in the ArgoCD namespace if the signature is missing.
  
  #Resources in argocd supervisor service

  # k get all -n svc-argocd-service-domain-c8
NAME                                                   READY   STATUS    RESTARTS   AGE
pod/argocd-service-controller-manager-69fd4477-mdvlf   1/1     Running   0          162m

  NAME                                                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/argocd-service-controller-manager-metrics-service   ClusterIP   ipaddr  <none>        8443/TCP   162m
service/argocd-service-webhook-service                      ClusterIP  ipaddr   <none>        443/TCP    162m

  NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-service-controller-manager   1/1     1            1           162m

  NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-service-controller-manager-69fd4477   1         1         1       162m
  
  

• After verification, deploy the ArgoCD instance YAML as per the documented sample configuration.
  
  #Basic argocd instance yaml
  
  apiVersion: argocd-service.vsphere.vmware.com/v1alpha1
kind: ArgoCD
metadata:
  name: argocd-1
  namespace: argoCD
spec:
  version: 3.0.19+vmware.1-vks.1

• Upon successful deployment, the expected ArgoCD pods should be created and running in the designated namespace.
  
  #Expected resources in argocd instance namespace for example here the names[ace created is argocd

  $ kubectl get all -n argocd
NAME                                      READY   STATUS      RESTARTS   AGE
pod/argocd-application-controller-0       1/1     Running     0          171m
pod/argocd-redis-86c8b4b5d7-wvtgk         1/1     Running     0          171m
pod/argocd-redis-secret-init-kjsg7        0/1     Completed   0          172m
pod/argocd-repo-server-7f8547f5f7-vwdxm   1/1     Running     0          171m
pod/argocd-server-56655bb478-bwvbz        1/1     Running     0          171m

  NAME                         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
service/argocd-redis         ClusterIP      ipaddr   <none>         6379/TCP                     172m
service/argocd-repo-server   ClusterIP      ipaddr   <none>         8081/TCP                     172m
service/argocd-server        LoadBalancer   ipaddr   ipaddr  80:32314/TCP,443:32141/TCP   172m

  NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-redis         1/1     1            1           171m
deployment.apps/argocd-repo-server   1/1     1            1           171m
deployment.apps/argocd-server        1/1     1            1           171m

  NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-redis-86c8b4b5d7         1         1         1       171m
replicaset.apps/argocd-repo-server-7f8547f5f7   1         1         1       171m
replicaset.apps/argocd-server-56655bb478        1         1         1       171m

  NAME                                             READY   AGE
statefulset.apps/argocd-application-controller   1/1     171m

  NAME                                 COMPLETIONS   DURATION   AGE
job.batch/argocd-redis-secret-init   1/1           56s        172m