Diego cell failing due to port 8888 conflict in EAR after Stemcell upgrade
search cancel

Diego cell failing due to port 8888 conflict in EAR after Stemcell upgrade

book

Article ID: 452710

calendar_today

Updated On:

Products

VMware Tanzu Platform Core VMware Tanzu Application Service

Issue/Introduction

  • Diego Cells in the CF deployment constantly flap from 'running' state to 'failing' state.
  • Execution of watch monit summary from an SSH on one of the failing Diego Cells shows  opamp-supervisor service in a restart loop.
  • Recreating the failing Diego Cells doesn't correct the failure, they enter failing state again shortly after recreation.

  • In EAR/TAS versions after 10.4, check logs in /var/vcap/jobs/opamp-supervisor/store/agent.log, you will see errors like:

    2026-08-17T18:03:37.383Z        error   [email protected]/service.go:165 error found during service initialization       {"resource": {"service.instance.id": "f7fb88bf-####-####-####-10e581f40b6e", "service.name": "otelcol-cf", "service.version": "0.11.34"}, "error": "failed to create meter provider: binding address localhost:8888 for Prometheus exporter: listen tcp 127.0.0.1:8888: bind: address already in use"}

  • Use the following command to identify which service is using port 8888 on all Diego Cells:

    bosh -d  <deployment> ssh diego_cell -c "sudo ss -tulnp"  | grep 8888

Environment

VMware Elastic Application Runtime (Tanzu Application Service) 10.4+

Cause

In VMware Tanzu Application Service (TAS) versions 10.4 and higher, the opamp-supervisor service (part of the OpenTelemetry/OTEL collector) uses port 8888 as the default egress port for the Prometheus Exporter. This port conflicts with third-party BOSH add-ons or tiles—specifically the Traceable-Agent tile—which also defaults to port 8888.

Resolution

To resolve the conflict, the port for the third-party agent must be changed, or the conflicting tile must be removed.

1. Verify Port Conflict

Run the following BOSH command to confirm which process is bound to 8888:

bosh -d <deployment> ssh diego_cell -c "sudo ss -tulnp | grep 8888"

Example output identifying traceable-agent: tcp LISTEN 0 4096 0.0.0.0:8888 0.0.0.0:* users:(("traceable-agent",pid=119231,fd=12))

2. Remediate Conflicting Agent

  • Option A (Reconfigure): Edit the runtime configuration for the Traceable-Agent to use a non-conflicting port.
  • Option B (Removal): If the agent is a POC or no longer required, remove the Traceable tile/add-on from the foundation and run Apply Changes.

3. Verify Service Recovery

Once the conflict is cleared, the opamp-supervisor should start automatically. You can manually trigger a restart if needed:

sudo monit restart opamp-supervisor

Additional Information

 

Powered by Wolken Software