Lifecycle Manager error: "A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot" when using proxy for the downloads.

Products

• VMware vCenter Server

Issue/Introduction

Error message on vCenter tasks:

"A general system error occurred: A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot"

 

 

vmware-vum-server.log:

YYYY-MM-DDTHH:MM:SS warning vmware-vum-server[10474] [Originator@6876 sub=VumVapi::Lib::Utils] [EmbeddedPyServiceProvider 472] Connecting to https://dl.broadcom.com/<downloadToken>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml failed, err: curl_easy_perform() failed: cURL Error: Failure when receiving data from the peer, CONNECT tunnel failed, response 407
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 686] Set CURLOPT_PROXY as https_proxy: PROXY_SERVER_FQDN:8080/
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 699] Set CURLOPT_NOPROXY as no_proxy: localhost, 127.0.0.1
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 744] Setup callback for SSL connections.
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * Hostname PROXY_SERVER_FQDN was found in DNS cache
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] *   Trying PROXY_SERVER_IP_ADDRESS:8080...
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * CONNECT tunnel: HTTP/1.1 negotiated
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * allocate connect buffer
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * Proxy auth using Basic with user '[email protected]'
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * Establish HTTP proxy tunnel to dl.broadcom.com:443
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] > CONNECT dl.broadcom.com:443 HTTP/1.1
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] Host: dl.broadcom.com:443
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] Proxy-Authorization: Basic U1JWQ19BMDA5QTNfUFJPWFlfUFJEQDYzc3RtYXJ5cy51ay53ZXN0cGFjLmNvbS5hdTpkb2dsZXhEaFpuQjVCNmloMVZ1TQ==
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] Proxy-Connection: Keep-Alive
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188]
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] < HTTP/1.1 407 Proxy Authentication Required
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] < Proxy-Authenticate: NTLM
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] < Server: BIG-IP
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] < Connection: close
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] < Content-Length: 93
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] <
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * Ignore 93 bytes of response-body
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * CONNECT tunnel failed, response 407
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 188] * closing connection #10
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 767] curl_easy_perform() failed: cURL Error: Failure when receiving data from the peer, CONNECT tunnel failed, response 407
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 767] [backtrace begin] product: VMware Update Manager, version: 8.0.3, build: build-24853646, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x########]
--> backtrace[01] libvmacore.so[0x########]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x########]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x########]
--> backtrace[04] libvci-vcIntegrity.so[0x########]
--> backtrace[05] libvci-vcIntegrity.so[0x########]: Sysimage::HttpDownloadToStream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Integrity::ProxyServer const&, Sysimage::HttpVerb, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::ostream&, int, bool)
--> backtrace[06] libvci-vcIntegrity.so[0x00E0D98B]: Sysimage::HttpDownload(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Integrity::ProxyServer const&, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int, b
ool, bool)
--> backtrace[07] libvci-vapi.so[0x########]: VumVapi::ServiceProvider::EmbeddedPyServiceProviderImpl::ValidateDepot(std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)
--> backtrace[08] libvci-vapi.so[0x########]: VumVapi::Lib::DepotManager::DepotContentManager::ValidateDepotContents(std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<Vapi::Data::ErrorValue>&)
--> backtrace[09] libvci-vapi.so[0x########]: VumVapi::Task::SyncDepotsTask::DoRun()
--> backtrace[10] libvci-vapi.so[0x########]: VumVapi::Task::AsyncTask::Run()
--> backtrace[11] libvmacore.so[0x########]
--> backtrace[12] libvmacore.so[0x########]
--> backtrace[13] libvmacore.so[0x########]
--> backtrace[14] libpthread.so.0[0x########]
--> backtrace[15] libc.so.6[0x########]
--> backtrace[16] (no module)
--> [backtrace end]
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[10474] [Originator@6876 sub=httpDownload] [httpDownloadPosix 741] Cleanup SSL context

 

 

Environment

vCenter Server Appliance 8.0.x

Cause

Error message "CONNECT tunnel failed, response 407" indicates authentication failure in the proxy server.

Resolution

1. Validate the username and password are correct.
2. Make sure the proxy defined on vCenter is configured properly to receive the username and password authentications, specially if the user are externally form the proxy server e.g. from AD
3. If proxy has a load balancer, make sure each of the nodes is correctly configured, use the IP address of each to validate.
4. Test the connection using curl command:
   curl -kvv -x "https://USERNAME%40DOMAIN.FQDN:PASSWORD@PROXY_SERVER_IPADDRESS:port" "https://dl.broadcom.com/<downloadToken>/PROD/COMP/ESX_HOST/main/vmw-depot-index.xml"