Standardizing Tanzu Application Service (TAS) Log Formats in Splunk via OTel

Products

• VMware Tanzu Platform Core

Issue/Introduction

In VMware Tanzu Application Service (TAS) environments, platform logs forwarded to external aggregators like Splunk may exhibit inconsistent formatting. While some components emit logs in JSON format (e.g., gorouter_stdout), others like access.log or standard syslog output remain in plain text. This inconsistency can complicate log parsing and indexing in downstream monitoring tools. The issue is typically discovered when users attempt to create unified dashboards or alerts in Splunk and find that field extraction fails for non-JSON log entries.

Environment

• Product: VMware Tanzu Platform Core / Tanzu Application Service (TAS)
• Components: Gorouter, OpenTelemetry (OTel) Collector
• External Integration: Splunk
• Versions: Not specified (General TAS behavior)

Cause

Log formatting is determined by the individual platform component and its specific application logic.

• JSON Format: Logs from gorouter_stdout are natively emitted as JSON by the gorouter application.
• Text Format: Logs such as access.log are emitted as plain text by the application. Broadcom does not mandate a single, unified log format across all platform components, and the OTel collector does not automatically convert all incoming log streams into JSON by default.

Resolution

Broadcom does not commit to delivering all platform logs in a native JSON format. However, you can achieve format consistency by using an OpenTelemetry (OTel) Transform Processor to parse plain text logs into structured fields before they reach Splunk.

To parse gorouter access logs into structured JSON-like attributes, follow these steps:

1. Access your OTel collector configuration.
2. Define a transform processor using the ExtractPatterns function with a regular expression matching the access.log format.
3. Apply the processor selectively using a where clause (e.g., filtering for appname == "gorouter").

Example Configuration Snippet:

processors:
  transform/parse_gorouter:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - |
            merge_maps(attributes, ExtractPatterns(body, "^(?P<router_host>\\S+)\\s+-\\s+\\[(?P<timestamp>[^\\]]+)\\]\\s+\"(?P<http_method>\\S+)\\s+(?P<http_path>\\S+)\\s+[^\\\"]+\"\\s+(?P<http_status>\\d+)\\s+(?P<bytes_received>\\d+)\\s+(?P<bytes_sent>\\d+).*vcap_request_id:\"(?P<vcap_request_id>[^\"]+)\".*response_time:(?P<response_time>[0-9.]+).*gorouter_time:(?P<gorouter_time>[0-9.]+).*instance_id:\"(?P<instance_id>[^\"]+)\""), "upsert") 
            where attributes["appname"] == "gorouter"

4. Add the transform/parse_gorouter processor to your logs pipeline.
5. Restart the OTel collector to apply the changes.

Additional Information

• Customization: Different regular expressions are required for different components (e.g., Diego cells vs. Gorouters) as log formats vary by component developer discretion.
• OTel Documentation: Refer to the official OpenTelemetry documentation for detailed syntax on the transformprocessor and available functions like merge_maps.