"trusted_certificates is invalid" error in Concourse after configure-director task fails

Products

• Concourse for VMware Tanzu

Issue/Introduction

The Concourse configure-director task from Platform Automation Toolkit fails with "trusted_certificates is invalid". For example,

task: configure-director

16:21:46  selected worker: 4b492511-####-####-####-26e######31 
16:21:47  4.4.10 
16:21:47  + vars_files_args=("") 
16:21:47  + ops_files_args=("") 
16:21:47  + om --env env/env.yml configure-director --config config/director.yml 
16:21:48  started setting iaas configurations for bosh tile 
16:21:50  finished setting iaas configurations for bosh tile 
16:21:50  started configuring director options for bosh tile 
16:21:51  2021/09/06 10:51:51 could not execute "configure-director": properties could not be applied: request failed: unexpected response from /api/v0/staged/director/properties: 
16:21:51  HTTP/1.1 422 Unprocessable Entity 
......
16:21:51  {"errors":{"security_configuration":["trusted_certificates is invalid"]}} 
 

Environment

Product Version: 5.5

Resolution

The property security_configuration.trusted_certificates in director.yml is set up to refer one credential in Credhub. 

For exampls case,  it is set to:  properties-configuration_security_configuration_trusted_certificates

However, it's found that the value of this credential is not stored properly and is not valid:

$ credhub get -n /concourse/myteam/properties-configuration_security_configuration_trusted_certificates
id: 03ade804-4d4e-44b1-ab10-7bfe284dc5f1
name: /concourse/myteam/properties-configuration_security_configuration_trusted_certificates
type: password
value: -----BEGIN CERTIFICATE-----  MIIDRzCCAi+gAwIBAgIUHjootWpUV4i9fA0F0g++s84+0NgwDQYJKoZIhvcNAQEL
  BQAwMzEf#########################################################UGl2b3RhbDAeFw0yMDA4MDQyMDI1NDNaFw0yNTA4MDMyMDI1NDNaMDMxHzAdBgNV
  BAMTFm9wc21nci1zZXJ2aWNlcy10bHMtY2ExEDAOBgNVBAoTB1Bpdm90YWwwggEi MA0GCSqGSIb3DQEBAQ##################################JWMvyX8ZN7Eu
  8URZA+2gb81nIoozq32weXMkTs#################mO9bvzG+zVIQ2zEl2qosi xe5yXuf6PIHF5mvbDZV5KquAif46JS9aS5TygQh9ck47mBUh2htH9nEyFXBUeTzJ
  ......
  ajmCLCa1id20fKeF##################bIpXjXM3JMhNvpp0p+OviWe9CkjqRn b/0QFt4h6YuL2tZ7XZkjAkEEtg2xoMHP9TDI
  -----END CERTIFICATE----- -----BEGIN CERTIFICATE-----
  MIIDHDCCAgSgAwIBAg################IwDQYJKoZIhvcNAQEL
  ...... 

To resolve this issue, delete this credential from Credhub:

$ credhub delete -n /concourse/myteam/properties-configuration_security_configuration_trusted_certificates
Credential successfully deleted

Set the credential again with a value in a valid format. For example,

$ credhub set -n /concourse/myteam/properties-configuration_security_configuration_trusted_certificates -t password -w "
-----BEGIN CERTIFICATE-----
> MIIDRzCCAi+gAwIBAgIUHjootWpUV4i9fA0F0g++s84+0NgwDQYJKoZIhvcNAQEL
> BQAwMzEfMB0GA1UEAxMWb3BzbWdyLXNlcnZpY2VzLXRscy1jYTEQMA4GA1UEChMH
> UGl2b3RhbDAeFw0yMDA4MDQyMDI1NDNaFw0yNTA4MDMyMDI1NDNaMDMxHzAdBgNV
> BAMTFm9wc21nci1zZXJ2aWNlcy10bHMtY2ExEDAOBgNVBAoTB1Bpdm90YWwwggEi
> MA0GCSqG############################G6qpKvSJ9LR7
> vpkRK0FNKII9nTEoQYLZcegYs3WdXGxRqhKcItu06herlyTIzKb1RUFyMZOXxOwk
> fRahS6b##############################wA1H1z53Osya
> ajmCLCa1id20fKeFAK23kJXq4jP3QS7fvobIpXjXM3JMhNvpp0p+OviWe9CkjqRn
> b/0QFt4h6YuL2tZ7XZkjAkEEtg2xoMHP9TDI
> -----END CERTIFICATE-----
> -----BEGIN CERTIFICATE-----
> MIIDHDCCAgSgAwIBAgIUT48V7a57OdfDpqXM8c9upjTLZ3IwDQYJKoZIhvcNAQEL
> BQAwFTETM#####################xaUayLmATrneoDLL3UV2
> YMB4URU1ysFGUMPNELGGSpzJZ3ywtPzOHqu+XbNR4+pciTu7xSavUZDfWuQ+Q3UZ
> tQDJ6uIq2iPJ0B###############################6URxkEQA0qHUWe5mmQ
> CKr/X2ImTXXNMGiWuO+N3O9HIxhgQbPY6ZRv4hqEjg8=
> -----END CERTIFICATE-----"

---------------------------------------------------------------------------------------

$ credhub get -n /concourse/myteam/properties-configuration_security_configuration_trusted_certificates
id: 03ade804-4d4e-44b1-ab10-7bfe284dc5f1
name: /concourse/myteam/properties-configuration_security_configuration_trusted_certificates
type: password
value: |-
  -----BEGIN CERTIFICATE-----
  MIIDRzCCAi+gAwIBAgIUHjootWpUV4i9fA0F0g++s84+0NgwDQYJKoZIhvcNAQEL
  BQAwMzEfMB0GA1UEAxMWb3BzbWdyLXNlcnZpY2VzLXRscy1jYTEQMA4GA1UEChMH
  UGl2b3RhbDAeFw0yMDA4MDQyMDI1NDNaFw0yNTA4MDMyMDI1NDNaMDMxHzAdBgNV
  BAMTFm9wc21####################################GX7hYwiv6os9iHAgMB
  AAGjUzBRMB0GA1UdDgQWBBT1EmkWTwzXuVwxgQdo3LsijFkZCzAfBgNVHSMEGDAW
  gBT1EmkWTwzXuVwxgQdo3LsijFkZCzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
  DQEBCwUAA4IBAQA37ilsNqJc1moxXpgDs9rNmVs9G9XaJHM9YRnCpvhXxuFTKhcz
  WoZ8O1fV4+h6i####################################Kb1RUFyMZOXxOwk
  fRahS6b8vCLJyIbDVtmNRC3JR21LTcVJ9p1SHVWlkgmVNawofqkRwA1H1z53Osya
  ajmCLCa1id20fKeFAK23kJXq4jP3QS7fvobIpXjXM3JMhNvpp0p+OviWe9CkjqRn
  b/0QFt4h6YuL2tZ7XZkjAkEEtg2xoMHP9TDI
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  MIIDHDCCAgSgAwIBAgIUT48V7a57OdfDpqXM8c9upjTLZ3IwDQYJKoZIhvcNAQEL
  BQAwFTETMBEGA1UEAxMKcG9zdGdyZXNDQTAeFw0yMTA4MTcxNDMzNDVaFw0yMjA4
  MTcxNDMzNDVaMBQxEjAQBgNVBAMTCWNvbmNvdXJzZTCCASIwDQYJKoZIhvcNAQEB
  BQADggEPADCCAQoCggEBAMPXFNtoW0tO/Gt67XEUx/sRSNgdqE8rv3v2ZRXIuQWh
  canl/mvOAfCdL7P9vIv###############################EB/wQCMAAwDQYJ
  KoZIhvcNAQELBQADggEBAImcko8m6ApnymYn84RkSwGUxaUayLmATrneoDLL3UV2
  YMB4URU1ysFGUMPNELGGSpzJZ3ywtPzOHqu+XbNR4+pciTu7xSavUZDfWuQ+Q3UZ
  tQDJ6uIq2iPJ0Bv1wUo5bWnfDexPNkwhqicmTkhl6LKVMteF9yh4O98DYFgXdqHc
  t4uy95QL1MNiL4LG72m4+jOglom0cKuAR7gZT/xLE4pOBojiSFt2AWUTGWmFKLE5
  Apof1bLsVDfWhjLoJjiDUG3GIU4mlQTZPTMR9KJHEdlHJb6URxkEQA0qHUWe5mmQ
  CKr/X2ImTXXNMGiWuO+N3O9HIxhgQbPY6ZRv4hqEjg8=
  -----END CERTIFICATE-----
version_created_at: "2021-09-23T11:49:56Z"

For information on an alternative way to set credentials through the CredHub API, refer to Credhub API - Set a Password Credential.