VCF 9.1 Component installation fails with error "component package URL not found, HTTP status:404 [VCFMS-PRECHECK-URL-003]" due to connectivity issue to offline Depot

Products

• VCF Operations

Issue/Introduction

This article speaks about the failure encountered when attempting to install VCF 9.1 Management or Instance components due to failure in fetching the binaries from the offline Depot. Below are some of the errors and related log snippets :

• Installing the components fail during the precheck phase with an error message similar to below:
  
  URL Connectivity Check Precheck. Status: FAILED.
The component package URL 'https://####/depot-service/content-gateway/PROD/COMP/<Component_Folder>/<Component_File>.tgz' was not found. HTTP status: 404. [VCFMS-PRECHECK-URL-003].
Please check the component package URL and try again. If the issue persists, contact support.
  
  
• The depot-service logs on Software-depot service, which is responsible for fetching and validating the binaries, show 'Depot Type: NONE' with HTTP status "404" entries (Refer to Additional Information section for steps to retrieve the depot-service logs) :
  
  vcf-fleet-depot/########-####/depot-service-########-####/file-server/console-log-########.log

YYYY-MM-DDThh:mm:ss stderr F DD/MMM/YYYY hh:mm:ss +0000 client="####.####.####.####" user_agent="curl/8.19.0" request="HEAD /depot-service/content-gateway/PROD/COMP/<Component_Folder>/<Component_File>.tgz HTTP/1.1" status="404" reason="SECURITY_EVENT"
YYYY-MM-DDThh:mm:ss stdout F ####.####.####.#### - - [DD/MMM/YYYY hh:mm:ss +0000] depot_type="NONE" "HEAD /depot-service/content-gateway/PROD/COMP/<Component_Folder>/<Component_File>.tgz HTTP/1.1" 404 0 "-" "curl/8.19.0" rt=0.000 uct="-" uht="-" urt="-" cache_status="-" cache_key="HEAD/depot-service/content-gateway/PROD/COMP/<Component_Folder>/<Component_File>.tgz" upstream="-" upstream_status="-"
YYYY-MM-DDThh:mm:ss stderr F DD/MMM/YYYY hh:mm:ss [info] ####: *####[lua] access_by_lua(nginx.conf:370):6: Request URI: /PROD/COMP/<Component_Folder>/<Component_File>.tgz - Depot Type: NONE, client: ####.####.####.####, server: content-gateway, request: "HEAD /depot-service/content-gateway/PROD/COMP/<Component_Folder>/<Component_File>.tgz HTTP/1.1", host: "<fqdn>"

Environment

VCF 9.1

Cause

The depot service logs explicitly flag Depot Type: NONE which shows software depot service is operating in disconnected mode without the required component binaries manually uploaded, causing the URL connectivity precheck to fail when attempting to fetch the component package. This was caused as the offline depot was not reachable over port 443 from VCF Management services IP pool where the Software Depot service pod resides

For more details about port requirements , refer to Port Requirements(filter Source with "Software Depot") 

Resolution

Re-configure the offline Depot after ensuring network firewall rules allow bidirectional TCP/443 traffic between the SDDC Manager, VCF Operations, VCF Management services network IP pool and the depot URL . 

For more details about how to configure depot, refer to Configure a Software Depot Connection Mode .

Additional Information

To retrieve the depot-service logs follow either of the below methods:

1. Offline method: Generate Support bundle for VCF Fleet depot service
   
   
   1. Login to VCF Ops using admin and navigate to Build > Lifecycle > VCF Management > Component > Software Depot > Actions > Generate support bundle and then Download support bundle
      
      
   2. Extract the bundle and navigate to  vcf-bundle-####-####_extracted/services-logs/vcf-fleet-depot/########-####/depot-service-########-####/file-server/console-log-########.log
      
      
2. Online method: Review the logs for file-server container inside depot-service pod:
   
   
   1. Login to one of the Mgmt services node using vmware-system-user


   2. Elevate to root:
      
      sudo -i
      
      
   3. Find the Control plane node:
      
      if [ -f /etc/kubernetes/manifests/kube-apiserver.yaml ]; then echo -e "\n This is a CONTROL PLANE node"; else echo -e "\n This is a WORKER node"; echo " Control Plane Endpoint: $(sudo grep 'server:' /etc/kubernetes/kubelet.conf 2>/dev/null | awk -F'/' '{print $3}')"; fi
      
      
   4. SSH to the control plane node using same user credential and set kubeconfig:
      
      export KUBECONFIG=/etc/kubernetes/admin.conf
      
      
   5. Check Fleet pods and the tasks running to obtain the depot-service container name :
      
      echo "--- FLEET PODS ---"; kubectl get pods -A | grep -E '(fleet|sddc)'; echo -e "\n--- COMPONENT TASKS (vmsp-platform) ---"; kubectl get tasks -n vmsp-platform --no-headers | grep -E '^(install|stage|configure)-component'
      
      
   6. From the above output, get the depot-service pod name and check the logs for file-server container inside it:
      
      kubectl logs -n vcf-fleet-depot depot-service-########-#### -c file-server |less