Argo-cd-notification-controller fails to trigger notification with "404 Not Found" due to token interpolation error.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

ArgoCD is failing to trigger GitLab notifications. The argo-cd-notification-controller pod logs display a "404 Not Found" error when attempting to execute a POST request to the GitLab API endpoint. 

argo-cd-notification-controller pod log:

YYYY/MM/DD HH:MM:SS [DEBUG] POST https://X.X.X.X/api/v4/projects/X/trigger/pipeline
{"level":"error", "msg":"Failed to notify recipient {gitlab-pipeline \"\"} defined in resource <ArgoCD-namespace>/<app>: request to {token=$gitlab-trigger-token
u0026ref=development\u0026variables [APP_NAME]=com POST https: //X.X.X.X/api/v4/projects/X/trigger/pipeline gitlab-pipeline} has failed with error code 404
: {\"message\": \"404 Not Found\"} using the configuration in namespace <ArgoCD-namespace>", "resource": "<ArgoCD-namespace>/<app>", "time": "YYYY-MM-DDTHH:MM:SSZ"}

Environment

vSphere with Tanzu

Cause

The argocd-notifications-cm ConfigMap uses incorrect Go template syntax for the secret variable within the webhook template block. Because Kubernetes secrets natively store data as base64-encoded byte arrays, referencing the secret directly without explicit string formatting causes the Go template engine to pass the literal string "$gitlab-trigger-token", resulting in GitLab rejecting the authentication attempt.

Resolution

1. Connect to the Supervisor cluster and edit the ArgoCD notifications ConfigMap:

kubectl edit cm argocd-notifications-cm -n <namespace>

2. Locate the template.gitlab-pipeline-trigger block within the ConfigMap.

3. Modify the token variable within the body payload to use the Go template index and printf functions. Update the configuration to the following:

Current Configuration:

template.gitlab-pipeline-trigger: |
  webhook:
    gitlab-pipeline:
      method: POST
      body: |-
        token=$gitlab-trigger-token&ref=development&variables[APP_NAME]={{.app.metadata.name}}

Required Configuration:

template.gitlab-pipeline-trigger: |
  webhook:
    gitlab-pipeline:
      method: POST
      body: |-
        token={{ printf "%s" (index .secrets "gitlab-trigger-token") }}&ref=development&variables[APP_NAME]={{.app.metadata.name}}

4. Save the ConfigMap. The argocd-notifications-controller will automatically detect the configuration change and interpolate the correct base64-decoded token into the GitLab trigger payload.