cloud_controller Apply Changes errors on task pre-start with failed Jobs: cloud_controller_ng in VMware Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms

Cloud Controller (CC) fails to update instance during Apply Changes, task fails with pre-start scripts failed, and failed jobs: cloud_controller_ng.

Change Log

Task 751 | 21:33:12 | Updating instance cloud_controller: cloud_controller/a4d91718-9a70-474f-afe0-a10109c4683e (0) (canary) (00:01:56)
                   L Error: Action Failed get_task: Task 20cc3bb3-2326-4524-4ad6-f8f0bf624c8f result: 1 of 4 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: bpm, syslog_forwarder, bosh-dns.
Task 751 | 21:35:08 | Error: Action Failed get_task: Task 20cc3bb3-2326-4524-4ad6-f8f0bf624c8f result: 1 of 4 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: bpm, syslog_forwarder, bosh-dns.

Task 751 Started  Mon Dec 30 21:28:53 UTC 2019
Task 751 Finished Mon Dec 30 21:35:08 UTC 2019
Task 751 Duration 00:06:15
Task 751 error

cloud_controller/pre-start.stderr.log

[2019-12-30 21:34:41+0000] + seed_db
[2019-12-30 21:34:41+0000] + pushd /var/vcap/packages/cloud_controller_ng/cloud_controller_ng
[2019-12-30 21:34:41+0000] + chpst -u vcap:vcap /var/vcap/jobs/cloud_controller_ng/bin/seed_db
[2019-12-30 21:35:07+0000] rake aborted!
[2019-12-30 21:35:07+0000] Sequel::ValidationFailed: Error for shared domain name *.mesh.example.com: name can contain multiple subdomains, each having only alphanumeric characters and hyphens of up to 63 characters, see RFC 1035.
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/model/base.rb:1462:in `save'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/model/base.rb:254:in `create'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/app/models/runtime/shared_domain.rb:42:in `block in find_or_create'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/database/transactions.rb:245:in `_transaction'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/database/transactions.rb:220:in `block in transaction'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/database/connecting.rb:253:in `block in synchronize'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/connection_pool/threaded.rb:92:in `hold'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/database/connecting.rb:253:in `synchronize'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/gem_home/ruby/2.4.0/gems/sequel-5.16.0/lib/sequel/database/transactions.rb:186:in `transaction'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/app/models/runtime/shared_domain.rb:28:in `find_or_create'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/cloud_controller/seeds.rb:81:in `block in create_seed_domains'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/cloud_controller/seeds.rb:75:in `each'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/cloud_controller/seeds.rb:75:in `create_seed_domains'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/cloud_controller/seeds.rb:11:in `write_seed_data'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/tasks/db.rake:110:in `block (3 levels) in <top (required)>'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/cloud_controller/background_job_environment.rb:15:in `setup_environment'
[2019-12-30 21:35:07+0000] /var/vcap/data/packages/cloud_controller_ng/ce52a7add677e6d81755d0e256079c61d037e188/cloud_controller_ng/lib/tasks/db.rake:109:in `block (2 levels) in <top (required)>'
[2019-12-30 21:35:07+0000] /var/vcap/packages/cloud_controller_ng/gem_home/ruby/2.4.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
[2019-12-30 21:35:07+0000] /var/vcap/packages/ruby-2.4-r5/bin/bundle:30:in `block in <main>'
[2019-12-30 21:35:07+0000] /var/vcap/packages/ruby-2.4-r5/bin/bundle:22:in `<main>

Cause

This problem is related to malformed domain name values within the VMware Tanzu Application Service (TAS) for VMs tile. TAS for VMs only accepts domain names which adhere to RFC 1035. Setting a domain name in a TAS for VMs tile with a non-alphanumeric character or non-dash character prevents cloud_controller_ng from seeding.

For example: Setting an improper domain name within TAS for VMs tile.

• Under TAS > Networking - Service Mesh > External Domain
• Domain name value: *.mesh.example.com

The `*` character is invalid and should be removed.

Environment

Product Version: 2.6

Resolution

1. Review logs for cloud_controller/pre-start.stderr.log to confirm if error cloud_controller_ng fails with following error:

Sequel::ValidationFailed: Error for shared domain name

2. After confirming the error in logging, review the name of the domain. In this case, the task failed since the domain had a wildcard character set.

• Invalid Domain name: *.mesh.example.com

Remove any non-alphanumeric characters from domain name. RFC 1035 syntax rules specifies domain names can only contain alphanumeric characters and hyphens of up to 63 characters.

• Valid Domain name: .mesh.example.com