Troubleshooting VCF Depot Connection Issues

Products

• VMware Cloud Foundation

Issue/Introduction

Troubleshooting SDDC Depot connection issues.

Symptoms:
Unable to connect to the Online Depot from the SDDC UI.

Environment

VMware Cloud Foundation

Cause

This issue can be caused by several factors. Please check all of the potential causes in the workaround section.

Resolution

Verify if the environment is using a proxy:

SDDC UI -> Administration -> Proxy Settings

In 5.x environments the proxy config is now stored in the platform DB. 
/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform -c "select * from proxy_config"

                  id                  | proxy_enabled |    proxy_host    | proxy_port | creation_time | modification_time | transfer_protocol | proxy_authenticated
--------------------------------------+---------------+------------------+------------+---------------+-------------------+-------------------+---------------------
 db70a7e1-44d3-457b-bd58-23248c257fa1 | t             | <proxyfqdn> |       3128 | 1703705170905 |     1705599942206 | HTTP              | f
(1 row)

In 4.x environments the proxy config is stored in /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties



Potential causes:

1. LCM service is not running

• SSH into the SDDC Manager, elevate to root , and check the service: systemctl status lcm
• If lcm is not in an active state, start the service with systemctl start lcm
• Verify that the lcm-crash KB has been run in the environment 314663

2. Firewall blocking 443 traffic between SDDC Manager and depot.vmware.com

• Check that the port is open between SDDC Manager and the depot with the following command(Add proxy to syntax if applicable):

nc -zv depot.vmware.com 443 

Expected output:
vcf@sddc-manager [ ~ ]$ nc -vz depot.vmware.com 443
depot.vmware.com [173.222.196.26] 443 (https) open

• If blocked work with firewall team to ensure traffic is not being blocked over required port 443. 

3. Depot certificate is missing from the SDDC trusted certificate store. 

• Trust the depot.vmware.com root certificate with the following command. If the certificate is not present the command will prompt "Trust this certificate?". If the cert is already present it will return, " Certificate not imported, alias <depot_cert> already exists"

echo off  | openssl s_client -connect depot.vmware.com:443  2>&1 | openssl x509 -outform PEM 1> /tmp/depot.crt  && keytool -importcert -alias depot_cert -file /tmp/depot.crt -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

Note: If they're using a proxy:

Example (replace the proxy fqdn & port number):

echo off  | openssl s_client -proxy <proxy-fqdn>:<proxy port> -connect depot.vmware.com:443  2>&1 | openssl x509 -outform PEM 1> /tmp/depot.crt  && keytool -importcert -alias depot_cert -file /tmp/depot.crt -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit

4. Incorrect Depot credentials 

• Test connecting to the depot with the following curl command:

curl -kv https://depot.vmware.com:443/PROD2/evo/vmw/index.v3 -u customer_connect_username

curl -kv https://depot.vmware.com:443/PROD2/evo/vmw/index.v3 -u customer_connect_username --proxy proxyip:proxyport

• Test logging into the the customer connect portal if the credentials do not work. Recommend customer to open a non-technical support request to unlock account. KB 319000

5. Account is locked out.

• Reset account password to unlock the account with the following link forgot-password

6. SSL inspection configured on proxy or firewall

Symptoms:
Able to connect to the depot with the curl command. (Credentials are authenticating)
Able to connect to the depot with openssl (Verifies certificates are valid and port 443 is open)
Receive invalid user credentials when connecting depot in UI.

Resolution:
Work with customer's firewall/proxy team and allow the VMware certificates. 
Trust the signing certificate of the proxy into SDDC trust store.