Error :"The kubeconfig is invalid. Please reach administrator for valid kubeconfig." when backing up CSE based Kubernetes clusters in VCD

Products

• VMware Cloud Director

Issue/Introduction

• When backing up Kubernetes(K8S) clusters created with VMware Cloud Director Container Service Extension (CSE) using VMware Cloud Director Object Storage Extension (OSE) you see the error below:
  
  The kubeconfig is invalid. Please reach administrator for valid kubeconfig


• The Kubernetes is correctly configured and up and running.
  
  
• From the HAR logs, the following stacktrace can be seen:
  
  com.vmware.voss.error.VossException: The kubeconfig is invalid. Please reach administrator for valid kubeconfig.
 at com.vmware.voss.k8s.utils.ClusterUtils.validateClusterConnectivity(ClusterUtils.java:128)
 at com.vmware.voss.resource.k8s.service.impl.task.K8sClusterProtectAction.doAction(K8sClusterProtectAction.java:207)
 at com.vmware.voss.resource.k8s.service.impl.ClusterServiceImpl.enableClusterProtection(ClusterServiceImpl.java:627)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Environment

VMware Cloud Director 10.x

VMware Cloud Director Object Storage Extension 2.2.2

VMware Cloud Director Object Storage Extension 3.0

VMware Cloud Director Container Service Extension 4.x

Cause

This issue occurs when the OSE server can't setup network connection with Kubernetes cluster. 

Resolution

To resolve this issue check the connectivity between OSE and Kubernetes cluster as below:

1. Verify that the primary node's IP is reachable from the OSE backend server.
   
   
2. Ensure the port 6443 is opened on  the k8-cluster master node running the command below form the OSE server:
   
   telnet $ClusterIP 6443


3. Ensure the K8S master / worker nodes can reach OSE server over port 443. If OSE has load balancer, the target should be the load balancer.
4. Verify that the cluster nodes can reach the OSE endpoint URL. Run the following command to note the OSE endpoint URL.
   
   ose show
   
   
5. Verify if the OSE VM can kubectl to k8s cluster. If the results show any error, then it indicates problems with OSE and K8S cluster.

   1. Download the kube-config from VMware Cloud Director (VCD) to OSE VM, save it to vcd-k8s-kubeconfig

   2. Download the kubectl to OSE VM

   3. Run the command 'kubectl --kubeconfig ./vcd-k8s-kubeconfig get nodes'

6. Verify Kubernetes cluster has access to  github.com (20.233.83.145:443)
7. Check the OSE Network ports configuration