Security Recommendations for integrating Tanzu Elastic Application Runtime Open Telemetry collector with AWS CloudWatch

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

In order to enable integration between Elastic runtime OTEL ( open telemetry collector ) and AWS CloudWatch we would have to create and IAM policy like the one described here https://aws-otel.github.io/docs/setup/permissions and enable this for each VM in the bosh deployment. 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:DescribeLogGroups",
                "logs:PutRetentionPolicy",
                "xray:PutTraceSegments",
                "xray:PutTelemetryRecords",
                "xray:GetSamplingRules",
                "xray:GetSamplingTargets",
                "xray:GetSamplingStatisticSummaries",
                "ssm:GetParameters"
            ],
            "Resource": "*"
        }
    ]
}

Granting these permissions to each VM such as diego cells exposes risk to the CloudWatch instance given each vm will have permission to create log groups and streams.   This KB offers a recommended solution to this problem and does not provide the steps for implementation.

 

 

Resolution

It is recommended to use the open telementy gateway deployment pattern https://opentelemetry.io/docs/collector/deploy/gateway/ to minimize risk and allow for more control over CloudWatch permissions.  Operators could configure an AWS ADOT Collector solution like described here https://aws-otel.github.io/docs/getting-started/cloudwatch-metrics . The logs will then flow from Elastic Application Runtime through a load balancer that points to subset of ADOT VMs that have the IAM permissions to ingest logs and metrics into CloudWatch.