"Cannot validate proxy configuration. Proxy server didn't allow connection from VCF SDDC Manager to destination URL", Configuring proxy in SDDC Manager fails

Products

• VMware Cloud Foundation
• VMware SDDC Manager

Issue/Introduction

• Configuring the proxy in SDDC Manager fails with one of these messages
  • "Cannot validate proxy configuration. Proxy server didn't allow connection from VCF SDDC Manager to destination URL"
  • "Can't validate proxy configuration. Destination test URL https://depot.vmware.com returned unexpected HTTP Status. Call VMWARE Support."
  • "Can't validate proxy configuration. Proxy server was not reachable with specified host and port. Please check Proxy host, port and protocol are correct, and connectivity from VCF SDDC Manager to Proxy Server."


• This issue is observed on VCF SDDC Manager 5.2 / 5.2.1 / VCF 9.0.

Environment

VMware SDDC Manager 5.2
VMware SDDC Manager 5.2.1
VCF 9.0

Cause

This issue is caused due to recent change of authentication method in Broadcom depot dl.broadcom.com. During proxy configuration, VCF SDDC Manager try to validate proxy configuration by using it to connect to dl.broadcom.com (depot.vmware.com) and expect specific HTTP Status codes as response. Depot hosting changed this code from 401 AuthRequired for unauthenticated connection to 403 Forbidden or 404 Not found or code 500.

 

May see the following log messages in operationsmanager.log:

YYYY-MM-DDT20:26:23.740+0000 ERROR [vcf_om,a660a2387ec74456,6742] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7300-exec-5] [I6RUEJ] PROXY_HOST_NOT_REACHABLE Proxy validation error. Status: PROXY_CONNECTION_ERROR
com.vmware.evo.sddc.common.core.error.InvalidInputException: Proxy validation error. Status: PROXY_CONNECTION_ERROR

YYYY-MM-DDT20:26:23.735+0000 INFO  [vcf_om,a660a2387ec74456,6742] [o.a.h.c.h.i.c.HttpRequestRetryExec,http-nio-127.0.0.1-7300-exec-5]
 Recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when processing request to {tls}->http://111.22.333.444:8443->https://dl.broadcom.com:443
YYYY-MM-DDT20:26:23.738+0000 DEBUG [vcf_om,a660a2387ec74456,6742] [c.v.vcf.secure.http.ProxyValidator,http-nio-127.0.0.1-7300-exec-5]
 Error testing proxy at 111.22.333.444:8443 connecting to https://dl.broadcom.com. org.apache.hc.core5.http.NoHttpResponseException: dl.broadcom.com:443 failed to respond

YYYY-MM-DDT20:26:23.741+0000 DEBUG [vcf_om,a660a2387ec74456,6742] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-
7300-exec-5] Handler Error Response: {"errorCode":"PROXY_HOST_NOT_REACHABLE","arguments":["111.22.333.444","8443","HTTP",null,"https://dl.broadcom.com"],"message":"Can't validate proxy configuration. Proxy server was not reachable with specified host and port.","remediationMessage":"Please, check Proxy host, port and protocol are correct, and connectivity from VCF SDDC Manager to Proxy Server.","referenceToken":"I6RUEJ"}

 

SSL traffic is not allowed through the proxy, even after importing the proxy certificate to the SDDC trust stores.

Resolution

1. SSH in VCF Installer/SDDC Manager appliance
2. Change to root
   
   su -
   
   
3. Edit the file /etc/vmware/vcf/operationsmanager/application.properties
   
   vi /etc/vmware/vcf/operationsmanager/application.properties
   
   
4. Add following line
   
   proxy.configuration.validation.expected.http.statuses=401,403,404,500
proxy.configuration.validation.test.url=https://dl.broadcom.com
   
   
5. Save the file and exit the editor (use Esc key + :wq!)  
6. Restart operationsmanager service
   
   systemctl restart operationsmanager
   
   
7. Try to set proxy configuration again
   
   
   

The same behavior could be observed in cloud builder proxy validation. If failure is encountered in cloud builder during bringup the following steps need to be followed:

1. Edit the cloud builder application.properties file: 
   vi /opt/vmware/bringup/webapps/bringup-app/conf/application.properties
   
   
   
2. Add following lines
   proxy.configuration.validation.expected.http.statuses=401,403,404,500
proxy.configuration.validation.test.url=https://dl.broadcom.com
   
   
3. Restart bringup service
   systemctl restart vcf-bringup

 

Additional Information

VCF Authenticated downloads Configuration Update Instructions

How to import Proxy server certificate to SDDC manager trust store.

 

Note:

If the application.properties file is read only, set the permissions to be able to edit the file:

/etc/vmware/vcf/operationsmanager ]# ls -la
/etc/vmware/vcf/operationsmanager/application.properties
-r-------- 1 vcf_operationsmanager vcf  356 Jul 30 23:38 application.properties

/etc/vmware/vcf/operationsmanager ]# chmod u+w application.properties