Import a Proxy Server Certificate to VMware Skyline Collector
search cancel

Import a Proxy Server Certificate to VMware Skyline Collector

book

Article ID: 317820

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Why proxy server certificate import is required


Skyline Collector validates explicitly the connection to VMware Cloud and will fail if the connection is not trusted. This can happen in cases when the proxy replaces the VMware server certificate with its own generated certificate, in order to be able to inspect the traffic.


Symptoms:

When proxy server certificate import is required

The following symptoms are seen when the proxy certificate is preventing the Skyline Collector from communicating with the VMware Cloud:

  • Customer is using proxy (either transparent or explicit) which has TLS traffic inspection capabilities
  • Testing the network settings in Skyline UI returns an error (i.e. cannot reach update site and/or cannot reach VMware Cloud)
  • The following error messages can be seen in the Skyline Collector logs: "Couldn't perform test registration due to: UploadException: Failed to upload due to exception >>> HttpRequestException: Certificate is not trusted"


Resolution

How to know if proxy certificate is being presented to the collector instead of VMware certificate

From Skyline Collector UI and console (SSH)

  • Login to Skyline collector
  • If already registered, go to Configuration → Network
  • If not registered, start the registration wizard until you reach Network configuration
  • Provide proxy settings (if required) and test the network connectivity
  • If an error is displayed, login to Skyline Collector via console (SSH)
  • Open the log file /var/log/skyline/api.log  and check at the bottom of the log file for any error messages that contain the string "Couldn't perform test registration due to: UploadException: Failed to upload due to exception >>> HttpRequestException: Certificate is not trusted"
  • If such messages are present, the proxy certificate is not trusted by the collector and needs to be imported

From Skyline Collector console (SSH)

If using transparent proxy:

 [ ~ ]# curl https://vcsa.vmware.com -v

If using explicit proxy:

[ ~ ]# curl -U [PROXY_USER:PROXY_PASSWORD] -x [PROXY_HOST:PROXY_PORT] https://vcsa.vmware.com -v

If proxy is not authenticated, the -U PROXY_USER:PROXY_PASSWORD should not be used
Note: Skyline Collector only supports HTTP proxies
If the above command returns the following error message:

"curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html
 
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above."

then the proxy certificate is not trusted by the collector and needs to be imported


Steps to import the proxy certificate

Retrieve proxy certificate(s)

This step is dependent on the proxy itself, and whether it is transparent or not. In the case of non-transparent proxies, please consult with documentation/network team on how to retrieve the proxy certificate

Transparent proxy certificate(s) retrieval

A transparent proxy is one that is forced upon the client without any modifications to the client proxy/network settings. In the case of Skyline Collector, a transparent proxy is not configured as a proxy in the Collector's configuration window, therefore the collector assumes the communication to VMware Cloud is direct, when in reality it always goes through the proxy.
To retrieve the proxy certificate (or certificate chain) from a transparent proxy, run the following openssl comand in Skyline collector console:

[ ~ ]# openssl s_client -showcerts -connect vcsa.vmware.com:443

An example response is given below:

CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = "VMware, Inc.", CN = vcsa.vmware.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Palo Alto/O=VMware, Inc./CN=vcsa.vmware.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
-----BEGIN CERTIFICATE-----
MIIGwjCCBaqgAwIBAgIQDbTyKj80XZ0zI2f4LMvNeDANBgkqhkiG9w0BAQsFADBP
.....
cn+BCd4w
-----END CERTIFICATE-----
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
-----BEGIN CERTIFICATE-----
MIIEvjCCA6agAwIBAgIQBtjZBNVYQ0b2ii+nVCJ+xDANBgkqhkiG9w0BAQsFADBh
.....
A7sKPPcw7+uvTPyLNhBzPvOk
-----END CERTIFICATE-----
 2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
.....
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Palo Alto/O=VMware, Inc./CN=vcsa.vmware.com
issuer=/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4410 bytes and written 393 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 6258E4F2B79D58D74B217C331FE5005A1FD1973C0242F18E6D93566FEE75506F
    Session-ID-ctx:
    Master-Key: D97A16CDC03B7A04A300129448539A0C701E95ADDF614E5986B82CA119C0960AE2F72F77DB9DB0B5F17712F63DEADD12
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1678382182
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

Copy the certificates from the response to separate files. The certificate is a Base64-encoded string put between the labels -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- . Note: copy both the encoded  parts and the labels to file.
From the example above, the following 3 certificates are identified:

[ ~ ]# cat cert1.pem
-----BEGIN CERTIFICATE-----
MIIGwjCCBaqgAwIBAgIQDbTyKj80XZ0zI2f4LMvNeDANBgkqhkiG9w0BAQsFADBP
.....
cn+BCd4w
-----END CERTIFICATE-----
[ ~ ]# cat cert2.pem
-----BEGIN CERTIFICATE-----
MIIEvjCCA6agAwIBAgIQBtjZBNVYQ0b2ii+nVCJ+xDANBgkqhkiG9w0BAQsFADBh
.....
A7sKPPcw7+uvTPyLNhBzPvOk
-----END CERTIFICATE-----
[ ~ ]# cat cert3.pem
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
.....
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
-----END CERTIFICATE-----
[ ~ ]# cat cert1.pem

As is the case above, the proxy returns a certificate chain. While it is not mandatory to import all certificates of the chain, it is recommended to import the innermost certificates, i.e. the intermediary and/or root certificates.

Import the certificates to the collector trust store

Once the proxy certificate is retrieved and copied over to the collector VA disk, run the below command to import it:

[ ~ ]# keytool -importcert -file <path_to_certificate_file> -keystore /usr/local/skyline/ccf/config/generated/customercerts -storepass $(cat /usr/local/skyline/ccf/config/generated/certs_key) -alias "myproxy"

Note: the alias can be any string. Do not import the certificate in any other truststores (such as the default java trust store)


Using the example above, the following is the output from the command:

[ ~ ]# keytool -importcert -file /tmp/cert.pem -keystore /usr/local/skyline/ccf/config/generated/customercerts -storepass $(cat /usr/local/skyline/ccf/config/generated/certs_key) -alias "myproxy"
Owner: CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Issuer: CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Serial number: 83be056904246b1a1756ac95991c74a
Valid from: Fri Nov 10 00:00:00 UTC 2006 until: Mon Nov 10 00:00:00 UTC 2031
Certificate fingerprints:
     SHA1: A8:98:5D:3A:65:E5:E5:C4:B2:D7:D6:6D:40:C6:DD:2F:B1:9C:54:36
     SHA256: 43:48:A0:E9:44:4C:78:CB:26:5E:05:8D:5E:89:44:B4:D8:4F:96:62:BD:26:DB:25:7F:89:34:A4:43:C7:01:61
Signature algorithm name: SHA1withRSA (weak)
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
 
Extensions:
 
#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 03 DE 50 35 56 D1 4C BB   66 F0 A3 E2 1B 1B C3 97  ..P5V.L.f.......
0010: B2 3D D1 55                                        .=.U
]
]
 
#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]
 
#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_CertSign
  Crl_Sign
]
 
#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 03 DE 50 35 56 D1 4C BB   66 F0 A3 E2 1B 1B C3 97  ..P5V.L.f.......
0010: B2 3D D1 55                                        .=.U
]
]
 
 
Warning:
The input uses the SHA1withRSA signature algorithm which is considered a security risk. This algorithm will be disabled in a future update.
 
Trust this certificate? [no]:  yes
Certificate was added to keystore

If importing more then one certificate, repeat the command for each certificate.
If the message Certificate was added to keystore is shown in the output, the certificate was successfully imported.

Restart the collector services

The collector services must be restarted. To do so, run the following command in Skyline Collector console (SSH):

[ ~ ]# systemctl restart ccf-collector health-collector


Once restarted, go back to Skyline UI and validate if the connection now goes without issues.
Note if using curl commands above: repeating the curl commands will yield the same result, since the certificates were imported only in the collector service trust store.
 


Powered by Wolken Software