Error: reconcile deployment/cci-service (apps/v1) namespace...

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When installing the Consumption Interface into a vSphere Supervisor Cluster, the status is Error (1) and when you select (1), you see :
  

  Reason: ReconcileFailed. Message: kapp: Error: waiting on reconcile deployment/cci-service (apps/v1) namespace: svc-cci-service-domain-c#: Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "cci-service-<ID>" has timed out progressing.)).

• In vCenter's tasks you see:
  

  Cannot complete the operation. See the event log for details. Error downloading plug-in. URL is unreachable. status code: 502, reason phrase: Bad Gateway

Environment

• vSphere Supervisor
• vCenter 8.0.u3

Cause

The Consumption Interface installation files are downloaded from projects.packages.broadcom.com from your vSphere Supervisor's Workload network. If your Workload network is unable to communicate with projects.packages.broadcom.com over HTTPS (port 443) for any reason, the installation of the Consumption Interface will fail. Some reasons include:

• A firewall is preventing communication from the Workload network to projects.packages.broadcom.com:443
• The Workload network cannot resolve the DNS name projects.packages.broadcom.com
• The Workload network doesn't have a route to projects.packages.broadcom.com 

Resolution

Allow the vSphere Supervisor Workload network to connect to the following domain names over HTTPS (port 443):

• projects.packages.broadcom.com
• projects.registry.vmware.com
• jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com

If the Consumption Interface is an error state when networking changes above are put in place, you will need to perform the following steps to get the Consumption Interface service into a Configured state:

1. From the vCenter's vSphere Client menu, select Workload Management
2. Select the Supervisors tab
3. Select the Supervisor where you are installing the Consumption Interface
4. Select the Configure tab > Supervisor Services > Overview
5. On the Supervisor Services page, select Installed > Consumption Interface > Manage
6. Without making any changes select Next > Finish
7. The Consumption Interface service should soon go into a Configured state and be ready for use

Additional Information

To verify this issue:

1. SSH to one of the ESXi host in the VKS cluster
2. cd /var/run/crx/
3. There will be a folder named imgfetcher-###-######
4. cd imgfetcher-###-######
5. You will see entries in guest.log such as:
   

   [timestamp] time="[timestamp]" level=info msg="trying next host" error="failed to do request: Head \"https://projects.packages.broadcom.com/v2/vcf_cci_service/###-#########-##-#######/manifests/sha256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\": dial tcp ##.###.###.##:443: i/o timeout" host=projects.packages.broadcom.com

Note that the IP address referenced in guest.log may change so firewall rules must be based on the domain names in the Resolution and not IP addresses.