Operational Considerations and Best Practices for Enabling Container Network Traffic Metrics in Tanzu Application Service

Products

• VMware Tanzu Application Service
• Vmware Tanzu Platform - SM
• VMware Tanzu Platform

Issue/Introduction

The "Enable container network traffic metrics for app instances" option in Tanzu Application Service (TAS) provides valuable visibility into application-level traffic patterns by reporting rx_bytes (received bytes) and tx_bytes (transmitted bytes) for all application instances. These metrics are typically emitted as Gauge envelopes and are instrumental for calculating traffic utilization at the container level.

Resolution

While enabling this feature is generally considered safe for production environments, it introduces additional telemetry data volume that flows through the Loggregator system (Doppler and Log Cache).

• System Overhead: The individual metric emission is lightweight, originating from the rep process on Diego Cells. However, in large foundations with thousands of application instances, the aggregate volume can contribute to increased load on Loggregator components.
• Loggregator/Log Cache Load: If these components are under-provisioned, high volumes of telemetry data can lead to egress log loss.
• Scalability: Broadcom has observed successful deployments at scale. If you observe log loss or slow metric retrieval (e.g., Apps Manager or cf app showing 0% utilization), it typically indicates that the Loggregator system requires additional capacity.

To ensure stability and performance, we recommend the following best practices when enabling this feature:

• Capacity Planning: Before enabling the setting in production, review your current provisioning for the Loggregator and Log Cache instance groups. Ensure these are scaled appropriately to handle the increased volume of telemetry data.
• Deployment: Enabling this setting is performed via the TAS tile and requires an "Apply Changes" operation. This process may trigger rolling updates to platform components such as Diego Cells or Dopplers, depending on your TAS version and configuration.
• Post-Deployment Monitoring: Monitor the health of the Loggregator system closely after enabling the metrics.
• Verification: If metrics appear to be missing (reporting zero values), do not immediately assume a platform failure. Use the Firehose CF CLI plugin (cf nozzle) to listen to the firehose directly. This verifies if the metrics are flowing correctly without relying on the Cloud Controller or Apps Manager, which may have polling limitations.

Additional Information

Related Knowledge Articles

• Consuming logs/metrics through the external V2 Firehose API show reduced performance – Provides guidance on identifying and mitigating performance impacts when consuming high-volume log and metric data.
• Apps Manager and CF CLI show 0% for CPU/Memory/Disk metrics – Offers troubleshooting steps for cases where metric data is missing or reporting zero values.