CCI (Cloud Consumption Interface) service installation in VKS fails with Failed to resolve image: Http request failed. Code 400 i/o timeout

Products

• VMware NSX
• VMware vSphere Kubernetes Service
• VMware Aria Operations (formerly vRealize Operations) 8.x
• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When attempting to install the Cloud Consumption Interface (CCI) service within a VMware Kubernetes Service (VKS) environment, the process fails during the image pull stage. You may observe the following error in the supervisor service logs or host events:

 

<ESXI_Hostname>,net: Failed to resolve image: Http request failed. Code 400:  ErrorType(2) failed to do request: Head "https://projects.packages.broadcom.com/v2/vcf_cci_service/cci-supervisor-service/manifests/sha256:<ShaHash>": dial tcp <Resolved IP of projects.packages.broadcom.com>:443: i/o timeout

 

Environment

• VMware Kubernetes Service
• VMware NSX

Cause

This log indicates the image pull is failing due to connectivity. The exact network path may vary but on an NSX backed deployment the traffic will be from the supervisor control VM's eth1 segment that is created to the IP shown in the error message (which is the resolved IP of projects.packages.broadcom.com).

The full network path needs to allow and route for the egress IP range assigned to the VKS workload network.This can be an  environmental issue where an external firewall (northbound of the NSX Tier-0 Gateway) is blocking outbound traffic eventing communication with the Broadcom package repository on port 443.

Resolution

Investigate the datapath and ensure there is connectivity from the CCI segment to projects.packages.broadcom.com or the resolved IP as indicated in the error message.

To validate if you can reach image repository from Supervisor Workload network

root@################# [ ~ ]# curl --interface eth1 https://projects.packages.broadcom.com -vvv

*   Trying #.#.#.#:443...

In order to resolve the issue, allow access to repository from Frontend/Workload network gateway on your network or Proxy server if configured.
Once the configuration has been updated, validate using the same command (above).

Sample:

# root@############### [ ~ ]# curl --interface eth1 projects.packages.broadcom.com -vvv

Connected to projects.packages.broadcom.com (#.#.#.#) port 443 (#0)

 

In an NSX environment, a traceflow can also be performed from the supervisor control VM to the relevant IP using layer 2 traceflow.

Additional Information

For Supervisor setup with VDS also refer: vSphere Pod Traffic to ClusterIP Time-outs

If you're unable to implement the resolution's connectivity requirements to projects.packages.broadcom.com, you can implement an air-gapped solution to store the images locally. Reference VKS Deployment Guide for Air-Gapped Environments

Similar issue for Contour deployment - Enabling Contour service on Supervisor fails with error: Deployment is not progressing