Products

VMware Tanzu Application Service

Issue/Introduction

BOSH deployment fails When running apply changes on TPCF tile with the following errors found in the /var/vcap/sys/log/otel-collector/ logs on the failed vm.

Error: cannot start pipelines: listen tcp 127.0.0.1:9100: bind: address already in use
2025/08/21 13:55:22 collector server run finished with error: cannot start pipelines: listen tcp 127.0.0.1:9100: bind: address already in use

Cause

Port 9100 is the default ingress port for OTEL collector and is used by the local TPCF forwarding agents to ingest logs into the OTEL collector pipeline. At the time of writing this article the ingress port of 9100 is not configurable through the operations manager TPCF tile.

Some third party tiles or bosh releases may include, for example, the Node Exporter bosh release which uses port 9100. If you are concerned there might be a conflict before enabling OTEL collector or before upgrading Tanzu Application service to 6.0.20/10.2.3 you can look for other processes using port 9100 by running the following command on your bosh deployed vms

bosh -d  <deployment> ssh <instance_group> -c "sudo ss -tulnp"  | grep 9100

Resolution

In a future version of TPCF the OTEL collector ingress port will be configurable to help in cases like this.

For now the workaround is to identify the BOSH add-on that is conflicting with OTEL collector port 9100 and take one of the following workarounds to mitigate this issue.

Workaround #1

Disable OTEL collector feature in the TPCF tile. This will stop the OTEL collector ingress process from starting freeing up port 9100 for the bosh add-on. This only works for TPCF 6.0.19 and lower or 10.2.2 and lower. TCPF 6.0.20 and 10.2.3 no long support disabling OTEL collector.
- To disable go to TPCF Tile -> System Logging and uncheck "Enable experimental OpenTelemetry Collector features". Additionally you must clear all text from the "OpenTelemetry Collector Configuration" field. If you do not do both then OTEL collector may still get deployed during apply changes.

Workaround #2

Change the OTEL Collector Listening port Manually
- SSH into operations manager vm
- cd /var/tempest/workspaces/default/metadata
- Search for the tpcf tile metadata file that matches your current deployed version. The "name: cf" refers to Tanzu Application Service product and the "product_version" should match your currently deployed version
  - ```
  ~$ head *
  ==> 521cbbcf8fb7.yml <==
  ---
  name: cf
  product_version: 6.0.20
  minimum_version_for_upgrade: 4.0.0
  pivnet_filename_regex: cf-.*\.pivotal
  label: VMware Tanzu Application Service
  rank: 90
  metadata_version: 3.0.6
  original_metadata_version: '3.0'
  service_broker: false 
```
- Open this file in an editor and go to the bottom. There you will find the "runtime_configs" section and add property ingress.grpc.port to the otel-collector job. The example below show how to set this property. Your version of this file will look different as we have omitted a lot of the other configuration to make it more readable. Please do not delete any other existing configuration.
  - ```
  - name: otel-collector
    runtime_config: |
      addons:
      - name: otel-collector
        jobs:
        - name: otel-collector
          release: tanzu-otel-collector
          properties:
            enabled: true
            ingress:
              grpc:
                port: 9102
```
- From Opsman webui go to "Review pending changes" and click on "see changes" in the "VMware Tanzu Application Service" tile and confirm the diff output shows the port changes.
- Proceed to run apply changes.

Workaround #3

Change the BOSH add-on listening port so it does not conflict with OTEL collector. For example if the add-on is the node-exporter release the following runtime config will change the default port of 9100 to 9191.
- ```
addons:
  - name: node_exporter
    jobs:
      - name: node_exporter
        release: node-exporter
        properties:
          node_exporter:
            web:
              port: 9191 
```

Workaround #4

Disable the conflicting BOSH add-on

Enabling OTEL Collector in Tanzu Platform Cloud Foundry may result in port conflicts with existing BOSH add-ons

Article ID: 411942

Updated On: