Attempting to install Rubrik IO filter may fail with error "A specified parameter was not correct: vibUrl Certificate is not trusted for vibUrl"

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

Installing Rubrik IOFilter may return error "A specified parameter was not correct: vibUrl Certificate is not trusted for vibUrl: https://<Rubrik Server>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip"
A general system error occurred: Cannot download offline depot from 'https:/<Rubrik Server>/connector/rubiofilter-bundle-1.1.22-10EM.800.1.0.20613240.zip'.
/var/log/vmware/eam/eam.log:

YYYY-MM-DDTHH:MM:SS | INFO | vlsi | RouteProvider.java | 226 | [AgencyBase->validateAgentConfigs:730edb8638750751] Certificate verification will not be made. Obtaining the certificate and the thumbprint from the provided URL https://<Rubrik Server FQDN or IP>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip using Envoy.
YYYY-MM-DDTHH:MM:SS | INFO | vlsi | RouteProvider.java | 325 | [AgencyBase->validateAgentConfigs:<>] https://<Rubrik Server FQDN or IP>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip conversion to use dynamic remote connection route completed. Result: Route(routedUrl:http://localhost:1080/external-tp/http1/<Rubrik Server FQDN or IP>/443/<Rubrik Server certificate Thumbprint>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip, url:https://<Rubrik Server FQDN or IP>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip, pemCertificate:-----BEGIN CERTIFICATE-----
YYYY-MM-DDTHH:MM:SS | INFO | vlsi | OpId.java | 37 | [URLChecker->checkURL:<>] created from [AgencyBase->validateAgentConfigs:<>]
YYYY-MM-DDTHH:MM:SS | INFO | vlsi | URLChecker.java | 132 | [URLChecker->checkURL:<>] Perfoming checks to URL( http://localhost:1080/external-tp/http1/<Rubrik Server FQDN or IP>/443/<connector ID>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip) for accessibility

YYYY-MM-DDTHH:MM:SS | INFO | vlsi | RoutedHttpOpExecutor.java | 128 | [URLChecker->checkURL:<>] Checking http://localhost:1080/external-tp/http1/<Rubrik Server FQDN or IP>/443/<Rubrik Server certificate Thumbprint>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip for accessibility.
YYYY-MM-DDTHH:MM:SS | ERROR | vlsi | EsxAgentManagerImpl.java | 547 | Exception:
com.vmware.vim.binding.eam.fault.InvalidUrl: null
at com.vmware.eam.agency.impl.LegacyAgencyBase.checkURL(LegacyAgencyBase.java:1120) ~[eam-server.jar:?]
at com.vmware.eam.agency.impl.LegacyAgencyBase.validateAgentConfigs(LegacyAgencyBase.java:1102) ~[eam-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_422]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]

/var/log/vmware/envoy-sidecar/envoy-access.log

YYYY-MM-DDTHH:MM:SS info envoy[2525] [Originator@6876 sub=Default] YYYY-MM-DDTHH:MM:SS GET /external-tp/http1/<Rubrik Server FQDN or IP>/443/<>/connector/rubiofilter-bundle
-1.1.22-1OEM.800.1.0.20613240.zip 526 upstream_reset_before_response_started{remote_connection_failure,TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED} UF 0 1310 - 14 - - 127.0.0.1:51624 HTTP/1.1 - 127.0.0.1:1080 - - - ##.##.##.##:443 - -
YYYY-MM-DDTHH:MM:SS info envoy[2525] [Originator@6876 sub=Default] YYYY-MM-DDTHH:MM:SS HEAD /external-tp/http1/<Rubrik Server FQDN or IP>/443/<Rubrik Server certificate Thumbprint>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip
404 upstream_reset_after_response_started{remote_reset} UR 0 0 - 53 51 - 127.0.0.1:51644 HTTP/1.1 - 127.0.0.1:1080 ##.##.##.##:37226 HTTP/2 TLSv1.2 ##.##.##.##:443 - -

/var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDTHH:MM:SS error vpxd[PID] [Originator@6876 sub=Default opID=<>] [VpxLRO] -- ERROR task-344054 -- ########-####-####-####-############(########-####-####-####-############) -- IoFilterManager -- vim.IoFilterManager.installIoFilter: :vmodl.fault.InvalidArgument
> Result:
--> (vmodl.fault.InvalidArgument) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> invalidProperty = "vibUrl"
--> msg = ""
--> }
--> Args:
-->
--> Arg vibUrl:
--> "https://<Rubrik Server FQDN or IP>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip"
--> Arg compRes:
--> 'vim.ClusterComputeResource:domain-<ID>'
--> Arg vibSslTrust:

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[3484646] [Originator@6876 sub=httpDownload] [httpDownloadPosix 782]
[backtrace begin] product: VMware Update Manager, version: 9.0.2, build: build-25148086, tag: vmware-vum-server, cpu: x8
6_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0048395D]
--> backtrace[01] libvmacore.so[0x003730D8]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x003855E5]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System
::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x00E49553]
--> backtrace[04] libvci-vcIntegrity.so[0x00E499C3]
--> backtrace[05] libvci-vcIntegrity.so[0x00E49EBD]: Sysimage::HttpDownloadFile(std::__cxx11::basic_string<char, std::cha
r_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<ch
ar> > const&, Integrity::ProxyServer const&, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocato
r<wchar_t> > const&, int, int, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const
&, bool const&)
--> backtrace[06] libvci-vcIntegrity.so[0x00E44D03]: Sysimage::DownloadJobHandler::Download()
--> backtrace[07] libvmacore.so[0x002CFC04]
--> backtrace[08] libvmacore.so[0x002D550F]
--> backtrace[09] libvmacore.so[0x00462AEB]
--> backtrace[10] libc.so.6[0x000890C4]
--> backtrace[11] libc.so.6[0x001091AC]
--> backtrace[12] (no module)
--> [backtrace end]
YYYY-MM-DDTHH:MM:SSZ verbose vmware-vum-server[] [Originator@6876 sub=httpDownload] [httpDownloadPosix 75
6] Cleanup SSL context
YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[] [Originator@6876 sub=DownloadMgr] [downloadMgr 709] Execut
ing download job {} throws error: curl_easy_perform() failed: cURL Error: SSL peer certificate or SSH remo
te key was not OK, SSL certificate problem: unable to get local issuer certificate
YYYY-MM-DDTHH:MM:SSZ error vmware-vum-server[] [Originator@6876 sub=DownloadMgr] [downloadMgr 817] Downlo
ad failed for url: https://<Rubrik Server FQDN or IP>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip

Cause

In order to install Rubrik filter, vCenter Server Appliance (VCSA) performs a HEAD request to Rubrik Appliance. The HEAD request over HTTP2 fails from the server end resulting in the error.

Resolution

Workaround 1:

Add the the self signed certificate to the vCenter Certificates Store, please follow the below steps:

Login to VCSA through SSH using root.
Run the below command:

/usr/lib/vmware-updatemgr/bin/updatemgr-utility.py install-cert <Rubrik Server FQDN or IP>

Workaround 2 :

proceed to disable http2 request for Rubrik server communication

Identify the Rubrik Server hostname/IP
Log in to VCSA using ssh and execute the below command to identify the certificate thumbprint

zgrep "<Rubrik server hostname/IP>" /var/log/vmware/envoy-sidecar/envoy-access* | grep HEAD

Sample Output:

YYYY-MM-DDTHH:MM:SS HEAD /external-tp/http1/<Rubrik server hostname/IP>/443/<Rubrik Server certificate Thumbprint>/connector/rubiofilter-bundle-1.1.22-1OEM.800.1.0.20613240.zip 404 upstream_reset_after_response_started{remote_reset} UR 0 0 - 53 51 - 127.0.0.1:51644 HTTP/1.1 - 127.0.0.1:1080 ##.##.##.##:37226 HTTP/2 TLSv1.2 ##.##.##.##:443 - -

OR

Alternate command to capture the thumbprint via ssh session on VCSA

openssl s_client -connect <Rubrik server hostname/IP>:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -sha1 -noout -in /dev/stdin | cut -d '=' -f2 | tr -d ':'

Capture the Rubrik Server certificate Thumbprint from the output in Step 2
Download the attached json file and modify the values for <Rubrik server hostname/IP> (step 1) and <Rubrik Server certificate Thumbprint> from Step 2

Sample:

21 "prefix": "/external-tp/http1/<Rubrik server hostname/IP>/443/<Rubrik Server certificate Thumbprint>/"

43 "address": "<Rubrik server hostname/IP>",

53 "<Rubrik Server certificate Thumbprint>"

Upload the updated json file to VCSA under the below location using SCP. Refer to How to upload or download files to or from vCenter and ESXi hosts

/etc/vmware-rhttpproxy/endpoints.conf.d/

Restart the rhttpproxy service

kill -SIGHUP `pidof rhttpproxy`

Proceed to register Rubrik IOfilter by accessing the Rubrik Management Interface

Attachments

rubrik.json get_app