Kibana not showing logs for prod application that used to show up last month

Products

• DX Operational Intelligence

Issue/Introduction

This had been working back in October, but is now not working.

Previously, we had been able to view the logs for my-host1 GCP pod in kibana discover using:

ao_itoa_logs_*_9eSSSS84_6d6f_4837_XXXX_b9184fdsfsf25_*

host: my-host1*

Using those logs that were found previously, we were able to set up alerts that worked in both kibana and DX Dashboards.

However, currently for some unknown reason, we are unable to view the logs in kibana for our application GCP (host: my-host1*).  Other applications in the same namespace are showing up (host: ab*). The logs show up on the GCP UI console. So  we are not sure why it is not getting into kibana. Why those logs stopped showing up in kibana?

Environment

Release : 21.3

Component : CA DOI LOG ANALYTICS

Resolution

This issue we found was that previously it was not required to specify a log level in the filebeat sidecar and some changes on the AIOPS side or non-user application side looks to have made that a requirement.

So, we had to add in an environment variable to the filebeat sidecar:

- name: LOG_LEVEL
    value: "info"

And then the logs started showing up in kibana again.