Configure TCA to forward logs to Aria for Logs
search cancel

Configure TCA to forward logs to Aria for Logs

book

Article ID: 415950

calendar_today

Updated On:

Products

VMware Telco Cloud Automation VCF Operations/Automation (formerly VMware Aria Suite) VMware Telco Cloud Platform

Issue/Introduction

  • Configure log forwarding

Analyze with the TCA Content Pack for Aria Logs.

Environment

  • Aria for Logs / vRealize Log Insight (vRLI) 8.x
  • TCA 3.2, 3.4
  • TCP 3.x, 4.x, 5.x

Resolution

Configure system log forwarding for TCA-M and TCA-CP Appliances

The following set of logs will for forwarded:

  • Container logs
    /var/log/containers/*.log
  • Kubelet logs    
    systemd (kubelet.service)

TCA 3.2:

Run the following on both the TCA-M and TCA-CP:

  1. Log into the TCA-M / TCA-CP admin portal:
    http://<fqdn>:9443
  2. Go to Configuration > Log management
    1. Select Destination Type as VMware Aria Operations for Logs
    2. Enter the Host endpoint in the following format:
      https://<fqdn>:9543
    3. Click on SAVE.
  3. After a few minutes, the logs will be forwarded and can be viewed in Aria Ops for Logs

TCA 3.4:

  1. Log into the TCA Manager appliance:
    https://<TCA-M_IP>
  2. Go to Administration > Logs > Log management
  3. Enter a friendly name.
  4. Select a TCA-CP.
  5. Enter the Log Server Type as VMware Aria Operations for Logs
  6. Enter the Log Server URL in the following format:
    https://<fqdn>:9543
  7. Click on ADD to import the certificate.

Configure log forwarding for CAAS Management cluster, Workload cluster, and Workload pods

Configure syslog forwarding during initial management cluster deployment

  1. Log into the TCA-M:
    https://<TCA-M_IP>
  2. During the Management Cluster Deployment, the user can enter the Host and Port  in the Syslog Servers section of the Kubernetes Cluster Details Page
    Note: This is optional and can be edited at a later date.
  3. Enter the Host name, 514 for the Port, and select UDP as the Protocol

Add syslog forwarding to an existing management cluster

  1. Log into the TCA-M:
    https://<TCA-M_IP>
  2. Go to Infrastructure > CAAS Infrastructure
  3. Select the Management cluster
  4. Click on Edit cluster configuration
  5. Click on ADD next to the Syslog Servers section
  6. Enter the Host name, 514 for the Port, and select UDP as the Protocol
  7. Click on SAVE

Forward Workload cluster journal logs using the systemsettings addon

  1. Add the systemsettings addon to the Workload cluster
  2. Enter the Host name, 514 for the Port, and select UDP as the Protocol

Note: These configurations will forward the cluster's journal logs.

Forward pod logs from the Workload cluster using the the Fluent-bit Add-on:

  1. Add the the Fluent-bit Add-on to the workload cluster
  2. Enable the Use Reference Configs option
  3. Modify the config as needed. See TCA 3.2: Advanced Configuration for Fluent-bit Add-On
  4. Deploy the Fluent-bit Addon.

NOTE: The Fluent-bit configuration provides the user with more flexibility to manage which logs can be forwarded to the log server.

TCA Log Configuration Summary

  Component syslog setting Logs forwarded Required
1 TCA-M and TCA-CP Logs
  • Configure Log server
  • Internally Fluent Bit is used
  • TCA/TCA-CP pod logs    /var/log/containers/*.log
  • Kubelet logs    systemd (kubelet.service)
 Y
2 Management cluster
  • Add syslog server
  • Internally used syslog-ng systemd service
  • Journal logs   
 Y
3 Workload cluster
  • Systemsettings addon 
  • Internally used syslog-ng systemd service
  • Journal logs  
 N
4 Pods on Workload cluster
  • Fluent-bit Addon
  • Uses Fluent-bit
  • All configured logs (Container, Journal Logs,Audit logs..,)
 Y

 

Additional Information

TCA-M and TCA-CP configuration

  • Fluent-bit is used to forward the system logs. Once configured, a couple of Fluent-bit pods will be created up and configuration is stored in a ConfigMap:

kubectl -n fluent-system get cm fluent-bit -o yaml

    [SERVICE]
        Flush 10
        Daemon Off
        Log_Level info
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag logs_kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
        Buffer_Chunk_Size 64KB
        Buffer_Max_Size 128KB

    [INPUT]
        Name systemd
        Tag logs_host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

    [FILTER]
        Name kubernetes
        Match logs_kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        Kube_Tag_Prefix logs_kube.var.log.containers.
        Buffer_Size 512KB

    [FILTER]
        Name lua
        Match logs_*
        Script timestamp.lua
        Call set_timestamp

    [FILTER]
        Name rewrite_tag
        Match logs_kube.*
        Rule $kubernetes['pod_name'] (.*) logs_k8s.$kubernetes['namespace_name'].$0/$kubernetes['container_name'].$record_date.log false
        Emitter_Name pod_tag
        Emitter_Mem_Buf_Limit 50MB

    [FILTER]
        Name rewrite_tag
        Match logs_host.*
        Rule $_PID (.*) logs_kubelet.service/kubelet.$record_date.log false
        Emitter_Name kubelet_tag
        Emitter_Mem_Buf_Limit 50MB

    [FILTER]
        Name nest
        Match logs_k8s.*
        Operation lift
        Nested_under kubernetes
        Add_prefix k8s_

    [FILTER]
        Name nest
        Match logs_k8s.*
        Operation lift
        Nested_under k8s_labels
        Add_prefix k8s_

    [FILTER]
        Name nest
        Match logs_k8s.*
        Operation lift
        Nested_under k8s_annotations
        Add_prefix k8s_

    [FILTER]
        Name record_modifier
        Match logs_k8s.*
        Whitelist_key k8s_pod_name
        Whitelist_key k8s_namespace_name
        Whitelist_key k8s_pod_id
        Whitelist_key k8s_host
        Whitelist_key k8s_app
        Whitelist_key k8s_container_name
        Whitelist_key k8s_docker_id
        Whitelist_key k8s_container_hash
        Whitelist_key k8s_container_image
        Whitelist_key k8s_app.k8s.io/component
        Whitelist_key k8s_app.kubernetes.io/instance
        Whitelist_key k8s_app.kubernetes.io/managed-by
        Whitelist_key k8s_app.kubernetes.io/name
        Whitelist_key k8s_app.kubernetes.io/part-of
        Whitelist_key k8s_helm.sh/chart
        Whitelist_key k8s_cni.projectcalico.org/containerID
        Whitelist_key k8s_cni.projectcalico.org/podIP
        Whitelist_key k8s_cni.projectcalico.org/podIPs
        Whitelist_key k8s_kubectl.kubernetes.io/restartedAt
        Whitelist_key log

    [OUTPUT]
        Name file
        Match logs_k8s.*
        Mkdir True
        Format template
        Template {log}
        Path /logs/retained-logs

    [OUTPUT]
        Name file
        Match logs_kubelet.*
        Mkdir True
        Format template
        Template {MESSAGE}
        Path /logs/retained-logs

    [OUTPUT]
        Name forward
        Match logs_*
        Host ${FLUENTD_HOST}
        Port ${FLUENTD_PORT}

CAAS Management cluster configuration:

  • Syslog-ng is used to forward the logs. It is a systemd process.

  • Configuration of syslog-ng (cat /etc/syslog-ng/syslog-ng.conf)

source src_local { systemd-journal();};

destination dst_remote_0 { network("<network-name>" transport(udp) port(514) ip-protocol(4)); };
log { source(src_local); destination(dst_remote_0); };

CAAS Workload cluster configuration:

  • Syslog-ng is used to forward the logs. It is a systemd process.

  • Configuration of syslog-ng (cat /etc/syslog-ng/syslog-ng.conf)

source src_local { systemd-journal();};

destination dst_remote_0 { network("<network-name>" transport(udp) port(514) ip-protocol(4)); };

Check Fluent-bit configuration on a workload cluster:

kubectl get cm -n tanzu-system-logging   fluent-bit-config  -o yaml

Powered by Wolken Software