Scaling Gorouter errors with IP already taken when assigning a Static IP

Products

• Operations Manager

Issue/Introduction

Symptoms:

When attempting to scale up Gorouter and assigning a static IP, Operations Manager Returns the following error for one of the assigned IPs:

IP "##.###.##.10" is already taken, can not be taken for 'cf-3923uh29f2992f3wf_router-9w8h9ehfwihfe9wifhw'
The following ips you requested for 'Pivotal Elastic Runtime - Router' can not be found in any assigned networks: ["##.###.##.10"]

The output of `bosh vms` command does not include an existing VM with the IP of##.###.##.10:

# bosh vms
+------------------------------------------------------------------------+---------+-----+-------------+---------------+
| VM                                                                     | State   | AZ  | VM Type     | IPs           |
+------------------------------------------------------------------------+---------+-----+-------------+---------------+
| router/0 (45caf45b-9ad4-4244-b96c-3bd4a7e137cc)                        | running | az1 | micro       |##.###.##.7  |
| router/1 (61625641-b326-434f-837e-4e3129eb4127)                        | running | az1 | micro       |##.###.##.8  |
| router/2 (b58cb6fd-ff8b-4143-9e2c-a8bee49ff3e4)                        | running | az1 | micro       |##.###.##.9  |
+------------------------------------------------------------------------+---------+-----+-------------+---------------+

This problem is not exclusive to Gorouter, you may see the problem with different job instance names listed. The resolution in this article should still apply.

Environment

Cause

In this case, the operator is attempting to assign IP address ##.###.##.10 to the new Gorouter, but Ops Manager  has already allocated that IP address to one of the errands. In this example, the notifications-ui errand holds the IP address.

We can check for this by reading through the installation.yml file.  There are multiple ways to do this, but if you're just reading the file the easiest way is to go to <example https://ops-man-fqdn/debug/files  example> This will display the installation.yml file, you can search for the IP in question or for the `ip_assignments` block.  See below for an example.

    "ip_assignments": {
        "assignments": {
            "cf-b726f387316441065827": {
                "notifications-ui-3f639252e9c255dade79": {
                    "a25903f1b4277d63b736": [
                        "##.###.##.10"
                    ]
                }

Resolution

The easiest resolution is to use a different and unused IP address for the new Gorouter.  If we can not choose a different IP address for the new Gorouter then we have to manually modify the Ops Manager configuration files to remove the IP from the errand VM. This will force Ops Manager to select a different IP address for the errand VM and assign the desired IP to the new Gorouter.

1. Use this KB How to Edit Operations Manager's Installation.yml to obtain an unencrypted copy of the installation.yml, edit the file as described in step #2 and then save and reencrypt the file.

2. Modify the IP assignments block for the errand that is using the desired IP address to include no entries. See below how there is an empty list now (i.e. `[]`).

    "ip_assignments": {
        "assignments": {
            "cf-b726f387316441065827": {
                "notifications-ui-3f639252e9c255dade79": {
                    "a25903f1b4277d63b736": []
                }

3. Repeat steps #1 and #2 for the actual-installation.yml file.  We need to modify both for this resolution.

4. Proceed to apply changes in Operations Manager and confirm the Gorouter is able to acquire the assigned IP address.

            notifications-ui-3f639252e9c255dade79:
        a25903f1b4277d63b736:
        -##.###.##.10

5.  Edit this entry and remove the ip assignment from the notifications-ui errand:

notifications-ui-3f639252e9c255dade79: 
​a25903f1b4277d63b736: []

6. Proceed to apply changes in Operations Manager and confirm the Gorouter is able to acquire the assigned IP address.