Updating Cluster with a Private Registry Fails: No such file or directory

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

When creating or updating a cluster with a Private Registry the worker node may fail to update. 

tkgi create-cluster --private-registries CONFIG-FILE

https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid-integrated-edition/1-20/tkgi/docker-custom-ca-certs.html

When looking at the specific task output, you will see that the update failed on the containerd job. In the containerd logs, you will find the below error:

/var/vcap/jobs/containerd/bin/containerd_ctl: line 104: /var/vcap/jobs/containerd/config/certs.d/#####/ca_#####.com/#####/#####.crt: No such file or director

 

Environment

TKGi v1.20

Cause

The cause of this failure is that the registry configuration path is in an incorrect format. On line 104 of the script, the containerd job builds the cert path like this:

ca_file_path=${PRIVATE_REGISTRIES_CERTS_DIR}/$server_dir/ca_$HOST-URL.crt

${PRIVATE_REGISTRIES_CERTS_DIR} = /var/vcap/jobs/containerd/config/certs.d

$server_dir = $REGISTRY-URL

 

Therefore, you can not have a nested path in your $HOST_URL.

 

Incorrect:

{
"servers": [
  {
    "url": "https://example.com,
    "hosts": [
      {
        "url": "https://docker-example.com/test/local,
        "capabilities": [
          CAPABILITIES
        ],
        "ca_cert": "-----BEGIN CERTIFICATE-----\nMIIFizC[...]\n-----END CERTIFICATE-----\n",
        "skip_verify": SKIP-VERIFY
      }
    ]
  }
]
}

Containerd configuration can only accept the hostname FQDN. Anything added to that (i.e., the path) will cause containerd to fail because it's expecting only the hostname FQDN. 

Resolution

Remove the nested path from the config file and re-run the command to update the cluster.

 

Correct:

{
"servers": [
  {
  "url": "https://example.com,
    "hosts": [
      {
      "url": "https://docker-example.com,
        "capabilities": [
          CAPABILITIES
        ],
        "ca_cert": "-----BEGIN CERTIFICATE-----\nMIIFizC[...]\n-----END CERTIFICATE-----\n",
        "skip_verify": SKIP-VERIFY
      }
    ]
  }
]
}

tkgi update-cluster --private-registries CONFIG-FILE