VS Code Explorer for Endevor "unable to verify the first certificate"

Products

• Endevor

Issue/Introduction

Recently deployed Endevor Web Services under a new trusted certificate using enterprise certificate authorities and find that this works fine in a browser.
Can access Endevor Services and Endevor WebUI and they report that the certificate is valid trusted.
However when try to update VS Code Explorer for Endevor connections the same URL is NOT deemed as trusted on Windows but the same version works under macOS.

The message seen in the VS Code when try to update the connection is as follows.
The current resolution is to allow untrusted certificates for Windows users.

Windows VS Code Output shows:

2024-07-09T22:26:10.084Z - [NBD3(I)] Edit an Endevor connection was called.
2024-07-09T22:26:10.086Z - Prompt for URL.
2024-07-09T22:26:16.420Z - Unable to validate Endevor Web Services certificate
2024-07-09T22:26:16.420Z - Unable to fetch Endevor Web Services API version because of error:
unable to verify the first certificate.
2024-07-09T22:26:16.424Z - Prompt for reject unauthorized option.
2024-07-09T22:26:31.437Z - Nothing was selected, continue with rejecting unauthorized connections.
2024-07-09T22:26:31.439Z - Prompt user to try again or continue.
2024-07-09T22:26:33.657Z - Try again was selected.
2024-07-09T22:26:33.657Z - Prompt for URL.
2024-07-09T22:26:35.686Z - Unable to validate Endevor Web Services certificate
2024-07-09T22:26:35.686Z - Unable to fetch Endevor Web Services API version because of error:
unable to verify the first certificate.
2024-07-09T22:26:35.690Z - Prompt for reject unauthorized option.
2024-07-09T22:26:38.600Z - Accept unauthorized connections was selected.
2024-07-09T22:26:38.601Z - Prompt for URL.
2024-07-09T22:26:40.360Z - Prompt for username.
2024-07-09T22:26:47.257Z - Prompt for password.
2024-07-09T22:26:51.023Z - Endevor connection NBD3 is updated.

The connection profile on the Mac is exactly the same as the Windows client.

Environment

VS Code Explorer for Endevor extension

Resolution

User tested with latest version of Explorer for Endevor and still same result.

Troubleshooting

Endevor Engineer had a long dialog with user with a series of troubleshooting suggestions

1. After initial investigations it seems it may be a problem with the intermediate certificate in the chain which is usually provided by the server itself when client is trying to establish the connection. The client uses that together with the root certificate to complete the chain and verify it all together. So it is clear that the VS Code extension platform does have different or, let's say, more strict verification compared with browser or on the system level, that makes it impossible to pass the verification in the VS Code environment. 


2. It seems that we were able to reproduce the issue in-house in our internal Endevor instances and it is related to the missing intermediate certificate in the chain of certificates reported by the server. For confirmation run this OpenSSL showcerts command to see the chain of certs that system receives from the WebServices instance: openssl s_client -showcerts -connect fqdn_hostname:8443 > openssl.out 2>&1
User ran the command on Mac (OpenSSL not available on Windows).
The results of the certificate chain verification show the chain is incomplete (verify error:num=21:unable to verify the first certificate) and probably have to be fixed on the Mainframe side to avoid the issue. Different network clients cope with this situation of missing intermediate certificates differently so it may explain why VS Code with the own nodejs runtime built-in responds to this case with the error.


3. In the previous messages the certificate chain reported by the Endevor Web Services instance was found using OpenSSL, which is a Unix-like OS way of dealing with certificates and a Windows approach can be slightly different. For that reason there is a purely Windows native utility certutil which can be used to run a certificate chain verification on Windows proceed with these steps:
a. Use the browser to download a certificate from the Web Services page, via the symbol to the left of the URL address. Use "Connection is secure" > "Certificate is valid" and on the the next dialog box the Details tab has an Export option. Save the file to a temp folder e.g. C:\temp with name certname.crt.
b. From command prompt cd to C:\temp and run the certutil command:
certutil -f -urlfetch -verify C:\temp\certname.crt > certutil.out

The certificate chain using the Windows utility seems to be complete and has no issues of locating the intermediate and root certs to complete the chain in comparison to the OpenSSL way that does not find intermediate cert right away.
The reason for this is that the Web Services instance responds with only the server certificate and not the intermediate one. This causes a failure to complete the chain when the OpenSSL tool chain is used for a certificate validation since OpenSSL, by the agreement, expects the intermediate cert to be provided by a server side. This is also what VS Code uses internally since it is developed with open source tooling.
On Windows, on the contrary, it is not restricted that way and both intermediate and root certs may be installed in and taken from the OS's certificate store so the server can respond with only server cert without intermediate. This is what usually happens in the enterprise system certification: the intermediate and root certs are installed by admins to the OS's secure stores or the browser's certificated DB's (Chrome for example).

To summarise: if it is possible, to avoid the confusion of some of the OpenSSL-based tools (like VS Code itself), it is recommended to ask the Endevor admin to update the server certificate with an intermediate too so it satisfies the both worlds: Windows and OpenSSL.
Another mystery that I could not solve is that why it does not complain when connecting with Explorer from Mac system, because in my tests it fails the same way as Windows on my Mac machine. Please provide the details of the connection that used in Endevor Explorer by hover over the connection node to see the details tooltip and can send a screenshot like this:





4. The info about the macOS connection shows it has certificate validation enabled and it is working fine, which is not successful for our test systems in house. But with assistance from our internal support team I managed to find a configuration that actually works on both Windows and MacOS machines. For that we made sure that both of the chain certificates (Root and Intermediate in this case) are properly installed in the system certificate store on both Win and Mac machines and this allows both certificate validation chains (OpenSSL and Win CertUtil) to validate an Endevor server certificate correctly. Again, it is more important that the OpenSSL verification process is working on the both systems because it is used by VS Code internally even though it does not explain the fact that the VS Code on Mac does not have any problems with the connection. 
At this point, I would recommend to try to ask the internal support team about their opinion on this situation and what else can be possibly done to overcome the issue with the OpenSSL certificate validation.


5. User then tested on Windows 11 and did not see the VS Code problem.
The in-house testing of certificates on Endevor setup and it was actually done with Win11 too. I am not certain though if it is related to the differences between Win10 and Win11 rather than configuration of the certificates but unfortunately I have no opportunity to test with Win10 since we do not have a Win10 machine in the team anymore.
Nevertheless, here are the details and screenshots of our setup and how we made it not reject the Endevor server certificate on all platforms.

First of all, our Endevor server responds with the only one certificate (of the server itself), the first in the chain even though that kind of a common agreement is to provide the server cert and all the intermediate ones by the server itself. That is close to the user's setup. Here is a snapshot of what Chrome shows when viewing the certs chain:

Now, a selected OS or browser should be configured to be able to go through the entire chain of the certificates (following the Issuer field) up to the very root cert, the initial issuer. For that we should make sure that both parent certificates (in our case) are installed somewhere in the OS (to be available for all applications in the system globally) or on the app/browser level (will be only available for this particular app and no others). The latter case is what describes the Chrome case the most, there is its own built-in certs store that can be used for that, but in this case obviously it would not be available for VS Code to utilize. As a result the Endevor server is considered "secured" only in the particular browser but VS Code in that case has no idea about the certs because it does not have access to the private certs store of the other app (Chrome). Just for reference Chrome's certs are available in Settings -> Privacy and Security -> Security -> Manage certificates area.

So we decided to go with the first option and our Global Support team made sure that it is installed in both OS'es, in the system certs stores.

For MacOS it is a Keychain Access app and the Root and Intermediate certs are both installed by the system admin:

For Windows it is the certmgr.msc app which can be found from the Start menu:

So from those all the apps on both OS'es that require to verify the certs have access to the system certs stores to restore the chain successfully even though the server itself responds with the only server certificate.

Another outcome of this also is that from now on everything that was signed with the same intermediate certificate, any other server certificate, will be automatically recognized as "secured" by all applications that work with the OS certs store for validation.

And again I do not see how this process depends on the Windows version itself, because it pretty much standard and should work similarly on Win10 and Win11, but maybe it will give some insights to check it with support team.

Conclusion

For this particular scenario there was no final resolution and the user continued to use the workaround of allowing untrusted certificates for Windows users.