vksm fails to auto-attach vks guest cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The auto-attach pod in Supervisor has the following below messages:
  
  { 
  "level": "error", 
  "time": "YYYY-MM-DDTHH:MM:SSZ", 
  "msg": "failed to attach cluster", 
  "controller": "vks-cluster-controller", 
  "controllerGroup": "cluster.x-k8s.io", 
  "controllerKind": "Cluster", 
  "Cluster": { 
    "name": "reallyfinal", 
    "namespace": "<namespace>" 
  }, 
  "namespace": "<namespace", 
  "name": "reallyfinal", 
  "reconcileID": "<id>", 
  "cluster": "<namespace>/reallyfinal", 
  "error_details": { 
    "status": "create cluster: failed to process response", 
    "rpc_error_code": "Internal", 
    "http_status": 500, 
    "response_body": { 
      "error": "Internal Server Error: please try again later", 
      "code": 13, 
      "message": "Internal Server Error: please try again later" 
    } 
  }, 
  
  
• The cluster-agent-service pod in VCF Automation has the following messages:
  
  {
  "component": "server-serve-grpc",
  "error": "projects.packages.broadcom.com/vsphere/vksm/extensions/9.0.2-0-25145732/tmc-bootstrapper/manifest:latest was not fetched: could not load image source: get image from the registry: Get \"https://projects.packages.broadcom.com/v2/\": dial tcp <ip-address>:443: i/o timeout",
  "grpc.method": "Create",
  "grpc.request.deadline": "YYYY-MM-DDTHH:MM:SSZ",
  "grpc.service": "vksm.private.v1alpha1.cluster.agent.AgentResourceService",
  "grpc.start_time": "YYYY-MM-DDTHH:MM:SSZ"
}

Environment

VMware vSphere Kubernetes Service

Cause

The cluster-agent-service in VCF Automation could not access the Broadcom public registry address projects.packages.broadcom.com. This mostly happens in air-gap environment or network connection is blocked by firewall. 

Resolution

• If the VCF Automation has the public network connection, need to check the underlying firewall to enable the accessing from cluster-agent-service to Broadcom public registry. More information about proxy, see How to Configure Proxy in VKSM Cluster-Agent-Service.
• If it is air-gap environment, follow the documentation Enabling VKS Cluster Management in an Air-Gapped Environment.

Additional Information

In air-gap environment, need to use script to download image from Broadcom public registry. The script need to specify the images list. The VCF 9.0.2 uses the below images:

/extensions/9.0.2-0-25145732/cluster-health-extension/manager:latest
/extensions/9.0.2-0-25145732/cluster-health-extension/manifest:latest
/extensions/9.0.2-0-25145732/cluster-sync-extension/cluster-sync-extension:latest
/extensions/9.0.2-0-25145732/agent-updater/agent-updater:latest
/extensions/9.0.2-0-25145732/cluster-sync-extension/manifest:latest
/extensions/9.0.2-0-25145732/agent-updater/manifest:latest
/extensions/9.0.2-0-25145732/agent-updater/agentupdater-workload:latest
/extensions/9.0.2-0-25145732/extension-manager/extension-manager:latest
/extensions/9.0.2-0-25145732/extension-manager/manifest:latest
/extensions/9.0.2-0-25145732/extension-updater/extension-updater:latest
/extensions/9.0.2-0-25145732/extension-updater/manifest:latest
/extensions/9.0.2-0-25145732/policy-insight-extension:latest
/extensions/9.0.2-0-25145732/gatekeeper:latest
/extensions/9.0.2-0-25145732/gatekeeper-operator/manifest:latest
/extensions/9.0.2-0-25145732/policy-insight-extension/manifest:latest
/extensions/9.0.2-0-25145732/gatekeeper-operator:latest
/extensions/9.0.2-0-25145732/intent-agent/intent-agent:latest
/extensions/9.0.2-0-25145732/intent-agent/manifest:latest
/extensions/9.0.2-0-25145732/fleet-mgmt/policy-sync-extension:latest
/extensions/9.0.2-0-25145732/tmc-bootstrapper/manager:latest
/extensions/9.0.2-0-25145732/fleet-mgmt/policy-sync-extension/manifest:latest
/extensions/9.0.2-0-25145732/tmc-bootstrapper/manifest:latest
/extensions/9.0.2-0-25145732/tmc-observer/tmc-observer:latest
/extensions/9.0.2-0-25145732/tmc-observer/logs-collector:latest
/extensions/9.0.2-0-25145732/tmc-observer/manifest:latest
/extensions/9.0.2-0-25145732/dataprotection/extension:latest
/extensions/9.0.2-0-25145732/dataprotection/manifest:latest