Datadog metrics missing for Otel Collector

Products

• VMware Tanzu Application Service

Issue/Introduction

After upgrade to EAR 10.2.10 with Datadog integration. It was discovered that the Datadog agent doesn’t have permissions to read the pid file of the otel-collector process in

/var/vcap/sys/run/bpm/otel-collector/otel-collector.pid

 

Environment

EAR 10.2.x problem have been discovered on 10.2.10 with stemcell upgrade. 

Cause

After successful upgrade it was discovered that otel-collector directory in /var/vcap/sys/run/bpm has permissions drwx------ whereas all the other directories have drwx-x-x

By Default (without Datadog Agent) all folders are with permission drwx------, There is a pre-start script that grants +x on the specified folders, and the otel collector folder was created after the pre-start script was executed leaving it with default drwx------ permissions.

Resolution

Short term resolution is to rerun the pre-start script.

bosh -d <DEPLOYMENT> ssh -c "sudo bash -x /var/vcap/jobs/dd-agent/bin/pre-start"

To verify 

bosh -d <DEPLOYMENT> ssh -c "sudo ls -axl  /var/vcap/sys/run/bpm"

Long term resolution is to update the dd-agent to run as post-start task this way all services will be running and the change will take place to all folders.