No event data from PKS / Kubernetes IP addresses in VR log insights (VMware Aria operations for Logs)

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

During normal operations the logs from TKGi API vms cannot be  found  in VR log insights (VMware Aria operations for Logs) from the Foundation1 TKGi servers

However same logs are visible for another foundation.

Both environments are configured as expected where the respective fields are correctly configured pointing to same VIP from VMware Aria operations for Logs

Environment

TKGi 1.2x

VMware Aria operations for Logs

Cause

Problem with user permissions preventing the logs to be visible from the VMware Aria operations for Logs UI, however if admin or user with admin privileges  is used the logs can be viewed.

In order to confirm the agent is correctly installed we used several different approaches to confirm.

1. Verify with logger if messages are recorded to the VIP on port 514 (basic syslog server) 

logger -d -n 172.xxx.xxx.10 -P 514 "<13>$(date '+%b %d %H:%M:%S') myclient curl-test: remote test message"      #UDP message on port 514
logger -T -n 172.xxx.xxx.10 -P 514 "<13>$(date '+%b %d %H:%M:%S') myclient curl-test: remote test message"      #TCP message on port 514
logger "<13>$(date '+%b %d %H:%M:%S') myclient curl-test: remote test message"                                                      #record message to localsyslog

2. verify if messages are sent from the tkgi-api vm verifying the conf file and running packet capture

bosh -d pivotal-container-service-<ID> vms
bosh -d pivotal-container-service-<ID> ssh pivotal-container-service/<ID>
sudo -i
cat /var/vcap/jobs/pks-vrli-control-plane-fluentd/config/fluent.conf    ### verify if VMware Aria operations for Logs is configured and the host and agent ID are present
<match **>
  @type loginsight_buffered
  host <FQDN>
  port 9543
  scheme https
    ca_file /var/vcap/jobs/fluentd/config/vrli_ca_cert.pem
    ssl_verify false
ping <FQDN of VRLI>                                                                                  ### confirm the IP
tcpdump -i eth0 host <IP of VRLI VIP>                                                     ### wait for some time to confirm if packets are flowing

3. Verify with tcpdump from the VMware Aria operations for Logs VIP and workers that packets are arriving

 

Resolution

If all these steps are confirmed to be in place and packets are received on the VMware Aria operations for Logs, further analisys might be required to confirm why the log messages are not visible.

User access permissions or filters taking place could prevent the messages to be displayed using the UI for more information follow the official documentation for additional details: https://techdocs.broadcom.com/us/en/vmware-cis/aria/aria-operations-for-logs/8-18/aria-operations-for-logs-8-18/managing-vrealize-log-insight-user-accounts/role-based-access-control.html