Upgrade fails with Error for shared domain name XXXX.com name overlapping

Products

VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
Deploy fails with error:

Task 206660 | 21:03:31 | Updating instance cloud_controller: cloud_controller/357490f9-a744-41e5-ae78-660429531c7f (0) (canary) (00:05:48) L Error: Action Failed get_task: Task 78fdd771-3648-422c-5367-bad9ff240dd8 result: 1 of 6 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: route_registrar, routing-api, bosh-dns, consul_agent, syslog_forwarder. Task 206660 | 21:09:19 | Error: Action Failed get_task: Task 78fdd771-3648-422c-5367-bad9ff240dd8 result: 1 of 6 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: route_registrar, routing-api, bosh-dns, consul_agent, syslog_forwarder. in the logs on the cloud controller :(/var/vcap/sys/log/cloud_controller_ng/pre-start.stderr.log)

Cloud Controller NG pre-start.stderr.log reports:

2018-11-28 17:46:41+0000] + chpst -u vcap:vcap /var/vcap/jobs/cloud_controller_ng/bin/seed_db [2018-11-28 17:46:46+0000] rake aborted!
[2018-11-28 17:46:46+0000] Sequel::ValidationFailed: Error for shared domain name subnet.example.com name overlapping_domain

Environment

Cause

Validation of overlapping domains was made more stringent in PCF version 2.2. This may cause upgrades to break if there are domains previously configured that disobey news rules for overlapping domains.

This problem occurs when you have a shared domain `subnet.example.com` underneath a private domain `example.com` The `example.com` domain is not shared, breaking a newer validation. The validation was added as a fix for a CVE. A domain that is owned by an organization can not be used by another organization.

Reference: https://www.cloudfoundry.org/blog/cve-2017-14389/

Resolution

It will be necessary to review existing domains and rectify overlapping domains. You may have trouble listing domains with Cloud Controller in this state, so it is necessary to address this via connecting to Cloud Controller database.

1. Connect to Cloud Controller database (CCDB) which is under PAS MySQL database. Perform steps in article: https://community.pivotal.io/s/article/How-to-Connect-to-the-PCF-Elastic-Runtime-Internal-MySQL-database

MariaDB []> use ccdb;

2. List out the domains. Review the domain names for overlapping domains (such as subnet.example.com which is under example.com in below output)

MariaDB [ccdb]> select * from domains;
+----+--------------------------------------+---------------------+---------------------+-----------------------------+----------+-----------------
| id | guid | created_at | updated_at | name | wildcard | owning_organization_id | router_group_guid | internal |
+----+--------------------------------------+---------------------+---------------------+-----------------------------+----------+-----------------
| 1 | 11bb1bb3-1282-47a3-ac7a-c519b00df2d2 | 2017-03-29 17:52:50 | 2018-08-20 20:56:24 | subnet.example.com | 1 | NULL | NULL | 0 |
| 19 | f3548479-6443-4d90-9110-d51ab880a7ad | 2018-08-22 13:06:56 | 2018-08-22 13:06:56 | example.com | 1 | 7813 | NULL | 0 |

3. Consider forcing example.com from private domain to shared in order to fix. This can be done by setting owning_organization_id to null which has affected of making domain shared.

MariaDB [ccdb]> update domains set owning_organization_id=NULL where id=19;

Note: This has effect of making domain shared across all PCF organizations, so make sure this is acceptable based on security policies.

Another solution to consider would be deleting or forcing subnet.example.com to private domain owned by same organization as example.com. Please contact Pivotal support if you are unable to find acceptable solution to address overlapping domains.

4. Retry apply changes and verify that CC pre-start script runs properly.