SSL/TLS server certificates specified in additionalTrustedCAs are not trusted in a RHEL-based VKS cluster.
Article ID: 448189
Updated On:
Products
Issue/Introduction
In a RHEL-based VKS cluster, even when you specify an SSL/TLS server certificate under additionalTrustedCAs as shown below, the certificate is not trusted on the actual worker nodes.
$ kubectl get cluster ##CLUSTER_NAME## -o yaml
...
- name: osConfiguration
value:
trust:
additionalTrustedCAs:
- caCert:
secretRef:
key: sample-ssl
name: sample-ssl-secret
...
If you are using a private registry for your Pod images, image pulling will fail with a "certificate signed by unknown authority" error, and Pods will fail to deploy.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 21m default-scheduler Successfully assigned default/sample-deployment-######## to ########
Normal Pulling 18m (x5 over 21m) kubelet Pulling image "registry.example.com/library/sample:v1"
Warning Failed 18m (x5 over 21m) kubelet Failed to pull image "registry.example.com/library/sample:v1": failed to pull and unpack image "registry.example.com/library/sample:v1": failed to resolve image: failed to do request: Head "https://registry.example.com/v2/library/sample/manifests/v1": tls: failed to verify certificate: x509: certificate signed by unknown authority
Warning Failed 18m (x5 over 21m) kubelet Error: ErrImagePull
Additionally, running a curl command on the worker node to access the host using that SSL certificate results in the following error:
$ curl -I https://registry.example.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.Environment
vSphere Supervisor
Cause
This issue occurs because the RHEL certificate installation process skips server (leaf) certificates.
When certificates are defined in additionalTrustedCAs, the VKS cluster places them under /etc/pki/ca-trust/source/anchors/machine-agent.crt on RHEL-based worker nodes, and then executes the update-ca-trust command to build the system-wide trust store. If the certificate does not have CA:TRUE in Basic Constraints extension, RHEL skips adding it to the trust store. Consequently, the server certificate is not trusted.
Resolution
To resolve this issue, specify the CA certificate instead of the server certificate in the Secret referenced by additionalTrustedCAs.
- Encode the CA certificate in base64 twice using the following command:
base64 -w 0 CA.crt | base64 -w 0
- Create a YAML file containing the double-encoded string, and apply it to the cluster:
apiVersion: v1
data:
##KEY_NAME##: |
<Insert the string generated in Step 1.>
kind: Secret
metadata:
name: ##SECRET_NAME##
namespace: ##NAMESPACE##
type: Opaque - Edit the cluster using kubectl edit cluster <cluster name> and reference the CA certificate Secret under spec.topology.variables:
- name: osConfiguration
value:
trust:
additionalTrustedCAs:
- caCert:
secretRef:
key: ##KEY_NAME##
name: ##SECRET_NAME## - Log into the worker node and verify that the target server is now successfully trusted:
curl -I https://server.example.com
Additional Information
Feedback
