"Exception occurred! Message: Missed Logs Details: 10000" message from Firehose Consumer Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The Firehose Consumer reports "Missed Logs"

Exception occurred! Message: Missed Logs Details: 10000

"Slow Consumer" may be reported in the Tanzu Application Service for VMs (TAS for VMs) Loggregator Traffic Controller logs, loggregator_trafficcontroller/loggregator_trafficcontroller.stderr.log:

2021/01/01 01:23:45.678901 Doppler Proxy: Slow Consumer from 10.1.2.3:45678 using /firehose/myfirehose

Querying Reverse Log Proxy Egress Dropped Messages shows high drop count.

A high historical value indicates the drops that happened over time. A better indication of an existing issue is if the counter is continuously increasing. Refer to "Scale indicator" in the Recommended thresholds section for more information. 

$ cf query ‘dropped{source_id=“reverse_log _proxy”, direction=“egress”}’ | jq .
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "deployment": "cf-abcd",
          "direction": "egress",
          "index": "00000000-0000-0000-0000-000000000000",
          "ip": "10.1.2.3",
          "job": "loggregator_trafficcontroller",
          "metric_version": "2.0",
          "origin": "loggregator.rlp",
          "product": "Pivotal Application Service",
          "source_id": "reverse_log_proxy",
          "system_domain": "system.mydomain.local"
        },
        "value": [
          1633596948,
          "47397917" <<<<<
        ]
      }
	]
  }
}

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Resolution

The "Missed Logs" message is generated by a callback method during the creation of the nozzle. (L61 - Code)

The details here indicate the number of logs that were "missed" while reading from the buffer. (CL121 - Code - one-to-one and L121 - Code - many-to-many)

The below diagram helps visualize what is happening: The code shown above is the client, it is running in the box code called "Consumer". The log buffer is in the box called "RLP: 8082".



The Consumer reads the buffer by chunks of 10000 logs. If the buffer fills up faster than the Consumer can read chunks, to ensure not dropping the newest logs, the oldest unread chunk is discarded (dropped). On the next iteration, the Consumer sees that the sequence number on the next available chunk available (the buffer), is higher than the last sequence number he had completed reading. The code then raises an error warning about how many logs were missed during the read.

Exception occurred! Message: Missed Logs Details: 10000

To remediate the issue, the Consumer side should be scaled out. More Consumer instances should be able to prevent the log buffer from filling up.

For more information on the baseline for scaling, refer to Reverse Log Proxy Egress Dropped Messages.

Once on the baseline, further scale up is possible if there's still drops.

Note: Another cause of this issue could be with the network bandwidth, check your network setting to rule this out.