Azure Log Analytics Nozzle sometimes crashes with Client Timeout Errors

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Microsoft Azure Log Analytics Nozzle for VMware Tanzu crashes with the following error in the Nozzle app logs:

2020-03-18T09:53:47.805-07:00 [APP/PROC/WEB/4] [OUT] {"timestamp":"1584550427.804669380","source":"oms-nozzle","message":"oms-nozzle.error posting message to OMS","log_level":2,"data":{"elapse time":"10.000476644s","error":"Post https://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.domain.azure.com/api/logs?api-version=2016-04-01: net/http: request canceled (Client.Timeout exceeded while awaiting headers)","event count":1000,"event type":"CF_LogMessage","remaining attempts":0,"total size":1095218}}
2020-03-18T09:53:47.868-07:00 [APP/PROC/WEB/4] [OUT] {"timestamp":"1584550427.867894411","source":"oms-nozzle","message":"oms-nozzle.Closing connection with traffic controller","log_level":2,"data":{}}

Resolution

In this connect, the Azure Firehose nozzle will connect to one of the traffic controller instances in the TAS Platform. If, for some reason, the traffic controller closes the connection, then the Azure Nozzle will crash and restart.

It is normal for the traffic controller to close the connection when a firehose nozzle is not keeping up with the traffic. Please refer to the firehose consumer metrics for more information on how to detect if this could be happening. 

This issue can also be caused by the customer's Azure Log Analytics tier being undersized for the log volume being produced by the foundation. This is especially likely if the customer is using a free or limited-quota tier of Azure Log Analytics. If the customer is using one of these tiers, they should consult with Microsoft about scaling their Azure Log Analytics subscription.

The log traffic will flow from the traffic controller -> Azure firehose Nozzle -> Microsoft Azure OMS (Operations Management Suite). In most cases, either the Nozzle is not able to keep up with the demand or the Downstream OMS is not able to keep up with the amount of traffic coming from the nozzles.