SDDC Manager Cluster Image Hardware Compatibility and Compliance Check Failed Due to vCenter Proxy Configuration

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms:

 

• During the cluster workflow we are getting an error on SDDC manager UI similar to:
  
  "Error (com.vmware.vapi.std.errors.error) => { messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => 
{ id = com.vmware.vcIntegrity.lifecycle.image.softwarespec.DownloadFailed, defaultMessage = 
Failed to download image from https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json, 
args = [https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json], 
params = <null>, localized = Failed to download image from https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827dbcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json }], 
data = <null>, errorType = ERROR }"
  
  
  
  
  
  
• SDDC manager contains errors similar to the excerpt below:
  
  /var/log/vmware/vcf/lcm/lcm.log
  2024-06-24T18:36:52.223+0000 ERROR [vcf_lcm,0000000000000000,0000,precheckId=0a7175d7-ad1b-4a9d-b630-4a4e371c42bc,resourceType=ESX_CLUSTER,resourceId=4e32d1a9-9d40-4874-aa06-02969e69f34b] [c.v.e.s.l.p.i.c.V
lcmPrimitiveImpl,Precheck-2] vLCM precheck failed due to:
com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = com.vmware.vcIntegrity.lifecycle.image.softwarespec.DownloadFailed,
defaultMessage = Failed to download image from https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json,
args = [https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json],
params = <null>,
localized = Failed to download image from https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json
}],
data = <null>,
errorType = ERROR
}
at com.vmware.vapi.std.errors.Error._newInstance2(Error.java:688)


• vCenter contains errors similar to the excerpt below:
  
  /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
  2024-06-24T20:36:52.091+02:00 info vmware-vum-server[250698] [Originator@6876 sub=DownloadMgr] [downloadMgr 668] Executing download job {140297522862848}, url=https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json
2024-06-24T20:36:52.091+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 619] Set CURLOPT_PROXY as https_proxy: http://XXX.XXX.XXX.111:3128/
2024-06-24T20:36:52.091+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 630] Set CURLOPT_NOPROXY as no_proxy: , localhost, 127.0.0.1
2024-06-24T20:36:52.091+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * Hostname XXX.XXX.XXX.111 was found in DNS cache
2024-06-24T20:36:52.091+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] *   Trying XXX.XXX.XXX.111:3128...
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * Connected to XXX.XXX.XXX.111 (XXX.XXX.XXX.111) port 3128 (#17)
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * CONNECT tunnel: HTTP/1.1 negotiated
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * allocate connect buffer
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * Establish HTTP proxy tunnel to vcf.hostname.domain.com:443
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] > CONNECT vcf.hostname.domain.com:443 HTTP/1.1
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] Host: vcf.hostname.domain.com:443
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] Proxy-Connection: Keep-Alive
2024-06-24T20:36:52.092+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181]
2024-06-24T20:36:52.199+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < HTTP/1.1 403 Forbidden
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Server: squid/4.10
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Mime-Version: 1.0
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Date: Mon, 24 Jun 2024 18:36:52 GMT
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Content-Type: text/html;charset=utf-8
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Content-Length: 3513
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < X-Squid-Error: ERR_ACCESS_DENIED 0
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Vary: Accept-Language
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Content-Language: en
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < X-Cache: MISS from webos2lpx002
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < X-Cache-Lookup: NONE from webos2lpx002:3128
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Via: 1.1 webos2lpx002 (squid/4.10)
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] < Connection: keep-alive
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] <
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * CONNECT tunnel failed, response 403
2024-06-24T20:36:52.200+02:00 verbose vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 181] * Closing connection 17
2024-06-24T20:36:52.200+02:00 error vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 685] curl_easy_perform() failed: cURL Error: Failure when receiving data from the peer, CONNECT tunnel failed, response 403
2024-06-24T20:36:52.202+02:00 error vmware-vum-server[250698] [Originator@6876 sub=httpDownload] [httpDownloadPosix 685] [backtrace begin] product: VMware Update Manager, version: 8.0.2, build: build-233199
93, tag: vmware-vum-server, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x0053D3AB]
--> backtrace[01] libvmacore.so[0x0043197E]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[02] libvmacore.so[0x00444252]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[03] libvci-vcIntegrity.so[0x00DA8C73]
--> backtrace[04] libvci-vcIntegrity.so[0x00DA9074]
--> backtrace[05] libvci-vcIntegrity.so[0x00DA98B0]: Sysimage::HttpDownloadFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::
char_traits<char>, std::allocator<char> > const&, Integrity::ProxyServer const&, std::__cxx11::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&, int, int, int, std::__cxx11:
:basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool const&)
--> backtrace[06] libvci-vcIntegrity.so[0x00DA4CBA]: Sysimage::DownloadJobHandler::Download()
--> backtrace[07] libvmacore.so[0x0037DBE6]
--> backtrace[08] libvmacore.so[0x003834F9]
--> backtrace[09] libvmacore.so[0x0051D093]
--> backtrace[10] libpthread.so.0[0x00008EAE]
--> backtrace[11] libc.so.6[0x000FDE2F]
--> backtrace[12] (no module)
--> [backtrace end]
2024-06-24T20:36:52.202+02:00 error vmware-vum-server[250698] [Originator@6876 sub=DownloadMgr] [downloadMgr 701] Executing download job {140297522862848} throws error: curl_easy_perform() failed: cURL Error: Failure when receiving data from the peer, CONNECT tunnel failed, response 403
2024-06-24T20:36:52.202+02:00 error vmware-vum-server[250698] [Originator@6876 sub=DownloadMgr] [downloadMgr 801] Download failed for url: https://vcf.hostname.domain.com/vmware/vcf/personalities/815952f2-c4e1-4e27-827d-bcda0e12d7cb/Content/SOFTWARE_SPEC_52f3b407-00eb-4bd6-1a00-f922e77b830a.json
  
  

Environment

VMware Cloud foundation 5.x

VMware Cloud foundation 4.x

Cause

An initial connection to the SDDC manager is established through vCenter, but when attempting to download the JSON file, the traffic is blocked by the proxy server.

Resolution

Verify the proxy server configuration for blocked traffic between vCenter and the SDDC manager.

Workaround:

1. 1. Connect to the vCenter Server with a SSH session and access the BASH.
      
      
   2. Modify the /etc/sysconfig/proxy file with a vi editor and add the SDDC manager FQDN and IP address to the following line, separated by a comma followed by a space character:
      
      NO_PROXY="localhost, 127.0.0.1, <SDDC_IP_ADDRESS>, <SDDC_FQDN>"
      
      For Example:
      
      NO_PROXY="localhost, 127.0.0.1, 10.10.10.100, vcf.hostname.domain.com"
      
      
      
   3. Apply the changes to the actual configuration using the following command on the vCenter:
      
      # export no_proxy="localhost, 127.0.0.1, <SDDC_IP_ADDRESS>, <SDDC_FQDN>"
      
      
      Note: Alternatively you can reboot the vCenter.
      
      
   4. Check the proxy settings are successfully applied using the command:
      
      # env |grep -i proxy


   5. Reattempt the compliance check

Additional Information

Note: vCenter 7.0.1 and later versions support specifying a CIDR notation (e.g., 1.2.3.4/24), netmask notation (e.g., 1.2.3.4/255.255.255.0), or a wildcard with a leading full stop (e.g., .*.vmware.com). Please note that wildcard entries must start with a full stop.

In that case, another workaround would be to include the full domain and network pool in the no_proxy configuration.

Both approaches should work.

For example:

NO_PROXY="localhost, 127.0.0.1, 10.10.10.0/24, .*.domain.com"