Apply Changes fails as Cloud Controller stops running after Update and jobs fail for log reviews on Reserved TCP Routing Ports

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

Your Apply Changes run consistently fails with an error similar to the following:

Task <Task#> | 20:20:42 | Error: 'cloud_controller/<guid> (instance)' is not running after update. Review logs for failed jobs: routing-api

You also have a Comma-Delimited list of reserved TCP Ports that you have configured in Operations (Ops) Manager for use with TCP Routing, Example:

Environment

Cause

This is a known-issue that currently affects all versions of Pivotal Cloud Foundry (PCF) that allow you to use TCP Routing. When using TCP Routing with a comma-delimited list of reserved ports configured in Ops Manager, the value in that field doesn't apply correctly to the Routing-API config on the Cloud Controller VM.

For example, if your Apply Changes failed with an error similar to the following:

Task <Task#> | 20:20:42 | Error: 'cloud_controller/<guid> (instance)' is not running after update. Review logs for failed jobs: routing-api

And on the Cloud Controller VM, you see errors similar to the following in /var/vcap/sys/log/routing-api/routing-api.log:

[2018-06-22 18:16:15+0000] {"timestamp":"1529691375.983918190","source":"routing-api","message":"routing-api.failed-load-config","log_level":2,"data":{"error":"Port must be between 1024 and 65535"}}

If you check the following file on the Cloud Controller, /var/vcap/jobs/routing-api/config/routing-api.yml, you will likely see your comma-delimited list of ports, without the commas in the reservable_ports field. Example:

    - name: routing-api
      properties:
        routing_api:
          router_groups:
+         - name: default-tcp
+           reservable_ports: 10258025
+           type: tcp

 

Resolution

As the Ops Manager does not support a comma-delimited list of ports, you can get around this issue utilizing the Routing API by performing the following steps:

Only enter one reserved TCP Routing port in Ops Manager, and Apply Changes so that this deployment succeeds.
Following the Guide for Modifying your TCP ports, you can run the following curl commands from the CF-CLI:

1. Get your guid for the TCP Routing Group:

cf curl /routing/v1/router_groups

2. Which will return JSON output similar to this: 

[
  {
    "guid": "9d1c1da9-ed63-45e8-45ee-256f8579455c",
    "name": "default-tcp",
    "type": "tcp",
    "reservable_ports": "10258025"
  }
]

3. Now that you have the guid, run the following command, inserting the proper guid and reservable_ports:

$ cf curl /routing/v1/router_groups/9d1c1da9-ed63-45e8-45ee-256f8579455c -X PUT -d '{"reservable_ports":"1025,8025"}'

4. This command will fix the reservable_ports using the Routing API, and will persist through future Apply Changes. You can verify that the change has applied by running the following command again:

cf curl /routing/v1/router_groups

Which should return the desired results:

[
  {
    "guid": "9d1c1da9-ed63-45e8-45ee-256f8579455c",
    "name": "default-tcp",
    "type": "tcp",
    "reservable_ports": "1025,8025"
  }
]