CloudSWG SSL interception and Google Cloud SDK SSL certificate verification failures
Article ID: 379096
Updated On:
Products
Issue/Introduction
httplib2 cannot reach <some.google.url>
Error: SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed. unable to get local issuer certificate (_ssl.c:1006)
gcloud init command fails with SSL CERTIFICATE_VERIFY_FAILED output
Error: Network connection problems may be due to proxy or firewall settings
Environment
CloudSWG with SSL interception enabled for Google domains
Google Safe search is enabled.
CASB gatelets are enabled, which causes SSL interception to occur for the Google domains of interest.
Cause
The Google Cloud SDK cacert.pem file located at C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\third_party\certifi does not contain the CloudSWG root certificate or CloudSWG intermediate cert in the certificate chain. This causes certificate verification errors to be returned by the Google Cloud SDK.
Resolution
There are several ways to remediate the problem. Please use the remediation solution that best fits your needs.
SOLUTION #1: Disable SSL interception for Google domains
Disabling SSL interception will prevent CloudSWG from doing SSL interception for Google domains. However, Safe search and Google specific CASB gatelets will be negatively impacted and will not function properly. This solution is provided in case you need a quick way to provide a solution for the end user.
To implement the SSL interception bypass, login to the portal (https://portal.threatpulse.com) and go to https://portal.threatpulse.com/#policyConfig|ssl and in the TLS/SSL Interception Policy, Add a rule that exempts google.com (and any other URL that may be returned with the certificate validation error) from being SSL intercepted. Click on the Activate Policy button to activate your changes.
SOLUTION #2: Disable SSL certificate validation within the Google Cloud SDK
To disable gcloud SSL certificate validation, please run the following command from the gcloud SDK command line interface:
gcloud config set auth/disable_ssl_validation True
The above command will disable SSL validation for any certificate encountered by the gcloud SDK, regardless of whether it is valid or not. Please note that the gcloud SDK will complain with every certificate that SSL validation is disabled. Here is a sample of the message being returned:
InsecureRequestWarning: Unverified HTTPS request is being made to host 'cloudresourcemanager.googleapis.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
SOLUTION #3: Add the CloudSWG root certificate chain to the cacert.pem file
This is the best solution, but may take some time to implement onto the local workstations. This solution adds the Cloud Services Root CA and Cloud Services CA - G2 to the cacert.pem file. As of the writing of this KB article, the location of the cacert.pem file is located at C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\lib\third_party\certifi. This location may change in the future, or it may be in a different location for other non-Windows operating systems. Please review Google's SDK documentation for the correct location of the cacert.pem file.
Here is what needs to be added to the top of the cacert.pem file:
============== <Please omit this line.> ==============
# Issuer: CN=Cloud Services Root CA O=BlueCoat Systems, Inc. C=US# Subject: CN=Cloud Services CA - G2 O=Blue Coat Systems, Inc. C=US# Label: "Cloud Services Root CA"# Serial: 130424312201033023037139521071936005756# MD5 Fingerprint: F1:89:B5:B7:CD:95:C5:57:10:00:4D:0A:EE:B3:0B:3A# SHA1 Fingerprint: 2B:4E:AF:4C:71:F5:F2:7F:BD:8F:0F:B5:5C:73:AB:C3:9F:15:14:7E# SHA256 Fingerprint: C3:B3:EE:7D:BF:E5:C3:A7:FC:F4:B2:F9:63:47:B2:45:4E:FE:9E:B1:9F:81:76:8F:DA:F9:C0:46:A9:A0:EC:A1-----BEGIN CERTIFICATE-----MIIDkjCCAnqgAwIBAgIQYh7PD8WR0TDUDVENFkFmfDANBgkqhkiG9w0BAQsFADBPMQswCQYDVQQGEwJVUzEfMB0GA1UEChMWQmx1ZUNvYXQgU3lzdGVtcywgSW5jLjEfMB0GA1UEAxMWQ2xvdWQgU2VydmljZXMgUm9vdCBDQTAeFw0xMTA5MDYwMDAwMDBaFw0zNjA5MDUyMzU5NTlaME8xCzAJBgNVBAYTAlVTMR8wHQYDVQQKExZCbHVlQ29hdCBTeXN0ZW1zLCBJbmMuMR8wHQYDVQQDExZDbG91ZCBTZXJ2aWNlcyBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxAB79qIpN0NApUS0be0NFYDqnY3g9jJsYZ6HVRsbw2eJnO2BKYhoBOW5fmUc9FaT0VbhIokHFRj4w3c2keWVgTlFHbp6EZaaK1H8yczTf57WlXILuCrJ9eGYsWE2doJePnFpT1QejDRQYMKTjAfQA0twCBSxxmZ5TzEJ/xAu4cYTc3CnMrgA3n+/tcH7Yn5PDNGAiwZMWf5OPbktH33b2r7yex+bgXXivY1Mw6k82RYLTLRsa8AoluBDTplqMbHo1QE7AuveeFkLL5GXX/8Uxao0mBvud2NJCHTZ9EcyHn5/Y2gnqJW4tmbMNXrrhAE+5Y1dWMAU8QFSF0aszQQE2wIDAQABo2owaDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAmBgNVHREEHzAdpBswGTEXMBUGA1UEAxMOTVBLSS0yMDQ4LTEtOTkwHQYDVR0OBBYEFKZKF9G8WLV3JRaSK9JMlSPPKBQ2MA0GCSqGSIb3DQEBCwUAA4IBAQCJszHQDBq6FlgoNRcgmgfn8LvyT1kWmBvM5UdZbPJwquKt4eqz67lXKzEnIUcWwdnJnkt0gmzXLw0zN5jwISiDbV5iGuJp6x+ftwwvHf9WxqM/aF9xQ9V5767GP4HCz0XfVcx0A1h+nJnh2suSISN6rPFhIhC5r/hbmBzzs/mjj60wFACDoP13Q2U3D+Jwm3Gf+LjQNHfLfPcBrJx9hKP8MJEDYPjHyLZTPd9keF3YfG5JevANWIK+4gzgbeVaLEV9/yXWRNEYxYhCy1nLwUcano2K8mgWkbUHctv7xw/SGymDCIrDnkHBrHqQ59YEfXWBZlLR0gyY56S1X7G8bD+o-----END CERTIFICATE-----
# Issuer: Cloud Services Root CA O=BlueCoat Systems, Inc. C=US# Subject: Cloud Services CA - G2 O=BlueCoat Systems, Inc. C=US# Label: "Cloud Services Root CA"# Serial: 125198044092137810583903576547706675314# MD5 Fingerprint: 91:F6:02:42:13:A1:FB:D2:1A:51:E3:5D:6C:3E:45:72# SHA1 Fingerprint: A2:5A:0C:5D:44:36:C0:0B:A2:8E:E7:FF:44:BD:6E:54:E5:AC:33:E3# SHA256 Fingerprint: 2D:26:0F:62:4D:B0:72:A1:60:41:3B:46:F5:07:75:2A:53:DD:C9:9C:A4:E8:C9:81:5D:91:EB:33:37:89:A6:BE-----BEGIN CERTIFICATE-----MIIEVTCCAz2gAwIBAgIQXjBEJk+PhZK1lccnVWogcjANBgkqhkiG9w0BAQsFADBPMQswCQYDVQQGEwJVUzEfMB0GA1UEChMWQmx1ZUNvYXQgU3lzdGVtcywgSW5jLjEfMB0GA1UEAxMWQ2xvdWQgU2VydmljZXMgUm9vdCBDQTAeFw0xNjAzMDQwMDAwMDBaFw0zNjA5MDQyMzU5NTlaMFAxCzAJBgNVBAYTAlVTMSAwHgYDVQQKExdCbHVlIENvYXQgU3lzdGVtcywgSW5jLjEfMB0GA1UEAxMWQ2xvdWQgU2VydmljZXMgQ0EgLSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALFLx79GVLTI+4KVALMpJ3Mopvfq+7FT9AH01soyLA8DNa/650Mk8Cgx6+Tz39CoTMdvePYhfPWdBD5U2PilzVY3QlkkalkNjlzv4+vvHgD1IwAohfpsAaxRnlR46LDYLpuolKVmsLr7LrDslAeVQzVq7X2t5nbaPQpKkttBsm2KtUwssApF/H16ORoV8WKFVsLU07gOdOEw0n7wv6TO5A2VYvF4mibwgE+YRJCZ45WhwypV5UrPRcqPgqm/pjvkV6zrrrx+rhcSeWO4TnX7vRjDOxwBU64N/DDjHwk9xElrnVajxuNGRBzxbwtopMBS1q3uVDpWAIU/rZFmQvJoKlsCAwEAAaOCASowggEmMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFTeW1hbnRlY1BLSS0yLTMzMTAdBgNVHQ4EFgQU9y03vNa3VaIBje4Ws9YiuFjXy0UwNwYIKwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vcGtpLW9jc3Auc3ltYXV0aC5jb20wXwYDVR0fBFgwVjBUoFKgUIZOaHR0cDovL3BraS1jcmwuc3ltYXV0aC5jb20vb2ZmbGluZWNhL0JsdWVDb2F0U3lzdGVtc0luY0Nsb3VkU2VydmljZXNSb290Q0EuY3JsMB8GA1UdIwQYMBaAFKZKF9G8WLV3JRaSK9JMlSPPKBQ2MA0GCSqGSIb3DQEBCwUAA4IBAQCHSSYYHe5mcewOGu1nyhcvyRAhLKLnkdMw7m/Rop04S+g5Qm6yIbQ1vNQf2MdHaHhJLsMacP/xp4/NUMHKW/kTImWnOLdRasyv/YY4W56duAiKZF8xrI1p/BiLqHzU2bJ5rnaaFUlmCvrlyvnwOCJ+2jfkPqEaoJje4D11wm/2cbWFTIVcJNS8v7iZlbUay57SK7UpCzLXqapMdJkJHAMVcDj+XHkfurKjMDx7jS25rKn1UU0wsLNciSrL7BE1VO6lPHf8m218XTvwLlJHlZGW5DOJBmEHCy5PRWL6euD+/Ldd6smHfpr6gKTnuMb+GlZNhjhSzTHbrFFiy0bmBIyt-----END CERTIFICATE-----
============== <Please omit this line.> ==============
Once the above contents have been added to the cacert.pem file, then the gcloud commands should work as expected. If the two certificates change, then the above data will need to change accordingly.
Additional Information
To get the certificate serial number, please use https://tools.keycdn.com/ssl to upload the certs and retrieve the serialNumber. The following openssl commands can be used to obtain the fingerprints:
openssl x509 -fingerprint -md5 -in Cloud\ Services\ CA\ -\ G2.crt
openssl x509 -fingerprint -sha1 -in Cloud\ Services\ CA\ -\ G2.crt
openssl x509 -fingerprint -sha256 -in Cloud\ Services\ CA\ -\ G2.crt
Potentially useful/interesting third party websites:
https://issuetracker.google.com/issues/249278577?pli=1
Feedback
