Error: "ImagePullBackOff" when installing fluentd for AKS (Cannot find repository)

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When deploying the Fluentd DaemonSet to collect logs, the Kubernetes pod fails to pull the required image, resulting in an ImagePullBackOff error. The issue is seen specifically when attempting to pull the image from the Broadcom registry:

  Warning  Failed     17s (x2 over 40s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    4s (x3 over 41s)   kubelet            Pulling image "projects.registry.vmware.com/vrealize_loginsight/fluentd"
  Warning  Failed     4s (x3 over 41s)   kubelet            Failed to pull image "projects.registry.vmware.com/vrealize_loginsight/fluentd": failed to pull and unpack image "projects.registry.vmware.com/vrealize_loginsight/fluentd:latest": failed to resolve reference "projects.registry.vmware.com/vrealize_loginsight/fluentd:latest": unexpected status from HEAD request to https://projects.registry.vmware.com/v2/vrealize_loginsight/fluentd/manifests/latest: 403

Environment

Aria Operations for Logs 8.18.x

Cause

Fluentd is a third party software, and there are many different way to install it.  There is no specific or precise way in going about to perform the installation.  In Aria Operations, the one way provided is to perform a "gem install fluentd".  As long as you can install fluentd successfully on the endpoint appliance, then it can be integrated with Aria Operations for Logs.
The legacy method involves installation of the additional fluentd-plugin-vmware-loginsight upon completing the installation of the fluentd. Currently, Operations for Logs supports fluentd built-in http format natively. There's nothing specific that should be done in order to have fluentd send logs to Operations to Logs server other than setting the endpoint of HTTP plugin to <fqdn>/api/v2/events.

Resolution

Once fluentd has been installed, the following is a sample template of configmap configuration for Kubernetes installation:

<source>
    @type tail
    path /var/log/syslog
    pos_file /var/lib/fluent/fluentd.log.pos
    tag system.syslog
    <parse>
        @type none
    </parse>
</source>

<filter **>
    @type record_transformer
    <record>
        text ${record["message"]}
    </record>
    remove_keys message
</filter>

<match **>
    @type http
    endpoint https://###.###.###.###:9543/api/v2/events
    tls_verify_mode none
    http_method post
    content_type application/json
    <format>
        @type json
    </format>
    json_array true
    <buffer>
        flush_interval 2s
    </buffer>
</match>

 

Precise configuration options can be found here:  Fluentd Output Plugin

Additional Information

In addition, review and modify the DaemonSet manifest to ensure the configmap is correctly mounted into the Fluentd container:

Modify the DaemonSet YAML to ensure the configmap is correctly mounted.

• Add the necessary volumes entry to the DaemonSet spec:

YAML
 
volumes:
- name: fluentd-config
  configMap:
    name: <Your-ConfigMap-Name>

• Add the corresponding volumeMounts entry to the container spec:

YAML
 
volumeMounts:
- name: fluentd-config
  mountPath: /fluentd/etc/fluent.conf
  subPath: fluent.conf

• Restart the DaemonSet pods for the changes to take effect:

Bash
 
kubectl rollout restart daemonset <daemonset-name> -n <namespace>