Troubleshooting cURL "error 60: SSL certificate problem: unable to get local issuer certificate" in vSphere Integrated Containers

Products

VMware vSphere ESXi

Issue/Introduction

Troubleshooting curl certificate errors inside vSphere Integrated Containers

Symptoms:

You see an error similar to the following when you try to connect to an external server from a container using the curl command:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Environment

VMware vSphere Integrated Containers 1.x

Cause

Transfer of sensitive information is typically done under the cover of digital certificates. The certificate will help confirm to the recipient that the sender is actually who they claim they are. Digital certificates are issued by certificate authorities or CAs. A list of trusted certificate authorities and their root certificates are installed on a server when a digital certificate is applied to the server. For transactions over HTTPS this information will be exchanged for communication. When a server CA is received via cURL that isn't signed by one of the trusted certificates in the installed CA certificate store, this will cause SSL to report an error ("failed to verify the legitimacy of the server") during the handshake. SSL will then refuse further communication with that server.

Resolution

If the insecure Certificate is a known public CA, attempt to re-download the curl ca-bundle directly from the curl site. You can use the following command:

curl --remote-name --time-cond cacert.pem https://curl.haxx.se/ca/cacert.pem

If the insecure certificate is a self-signed, known-trusted certificate, load the certificate to your local trusted certificate location for your OS.

Additional Information

https://curl.haxx.se/docs/sslcerts.html

Use an openssl or curl command similar to the following to identify the certificate that is untrusted:

openssl s_client -showcerts -connect vcsa1.example.com:443

Note: Replace vcsa1.example.com with the URL where the untrusted certificate is originating. You will see output similar to the following:

CONNECTED(00000003)
depth=0 CN = vcsa1.example.com, C = US
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = vcsa1.example.com, C = US
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=vcsa1.example.com/C=US
i:/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=vcsa1.example.com/OU=VMware Engineering
-----BEGIN CERTIFICATE-----
MIID9zCCAt+gAwIBAgIJAOZptRqIbAK+MA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD
VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ
FgVsb2NhbDELM........
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=vcsa1.example.com/C=US
issuer=/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=vcsa1.example.com/OU=VMware Engineering
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1499 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx:
Master-Key: 2184ABA9D7CEA14802EBFB1BECA6FE7DC322C76B9F9E6AAF82553DC5CA0B26F3C9A29342CD656CC47E5AF6B58661CA4A
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1585935973
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---

curl -kvvv https://vcsa1.example.com

Note: Replace vcsa1.example.com with the URL where the untrusted certificate is originating. You will see output similar to the following:

* Rebuilt URL to: https://vcsa1.example.com/
* Trying 192.168.0.51...
* TCP_NODELAY set
* Connected to vcsa1.example.com (192.168.0.51) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* ignoring certificate verify locations due to disabled peer verification
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=vcsa1.example.com; C=US
* start date: Apr 12 11:57:19 2019 GMT
* expire date: Apr 6 11:57:19 2029 GMT
* issuer: CN=CA; DC=vsphere; DC=local; C=US; ST=California; O=vcsa1.example.com; OU=VMware Engineering
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: vcsa1.example.com
> User-Agent: curl/7.59.0
> Accept: */*
>

Impact/Risks:

Connecting to an unknown insecure server can be hazardous. Only use the insecure flag in curl to a server if you can verify that the server is safe.
Adding an unknown CA to the trusted CA can be risky. Only add a CA to curl trusted CAs if you verify the legitimacy of the CA