How to backup Kubernetes clusters with version 1.24 and later by Cloud Director Object Storage Extension

Products

VMware Cloud Director

Issue/Introduction

This article illustrates a workaround to overcome this issue and makes the backup success by Object Storage Extension.

Symptoms:

When VMware Cloud Director user uses Object Storage Extension to backup Kubernetes clusters with version 1.24 or later, the backup will fail with an error.

500 internal {"status":500,"code":"InternalError","message":"Rollback bucket tenant-tkg220-bk due to failing to enable cluster protection urn:vcloud:entity:vmware:capvcdCluster:cf073c7c-2783-4338-a4e4-638fcbc33997, please contact the system administrator.","resource":"/api/v1/k8s/clusters/urn:vcloud:entity:vmware:capvcdCluster:cf073c7c-2783-4338-a4e4-638fcbc33997/protection","requestId":"da354f23-cea3-43e8-9818-a16e829775f2"}

In the Object Storage Extension server log, k8s-br-default.log shows that the deployer failed to get the secret from the service account.

"level": "error",
"timestamp": "2023-09-13T03:42:58.503-0400",
"caller": "service/cluster.go:69",
"msg": "Preparing cluster environments meet error with many times, give up.",
"times": 2,
"error": "can't find the secret name from the servie account"

In the Kubernetes cluster, you can find the namespace for Velero is created and the service account is created. But the service account ose-k8s-br has no secret.

> kubectl get ns | grep velero 

velero-5d440c9f-3fa7-4634-a1a3-2e0aa35110e7   Active   42m

> kubectl -n velero-5d440c9f-3fa7-4634-a1a3-2e0aa35110e7 get sa 
NAME                  SECRETS   AGE
ose-k8s-br            0         60m

As a result, the user fails to protect the Kubernetes cluster.

Environment

VMware Cloud Director 10.x

Resolution

After the cluster protection failure from Object Storage Extension, the cluster owner can create a service account secret and add it to the service account by kubectl command-line tool.

Tip

kubectl command-line tool supports to run in Windows, MacOS and Linux. To use kubectl to connect to your Kubernetes cluster, you need to download kubeconfig file to your local OS where kubectl is installed. If it's a Cloud Director Container Service Extension managed Kubernetes cluster the kubeconfig file can be downloaded as per the documentation, Kubernetes Container Clusters UI Plug-in for VMware Cloud Directo .
Organizing Cluster Access Using kubeconfig Files tells how to use kubeconfig with kubectl. The simplest way is to add parameter --kubeconfig <kubeconfig file path> in each kubectl command.
Make sure the OS you run kubectl has network access to the Kuberentes cluster's control plane node IP. For Cloud Director Container Service Extension managed Kubernetes cluster, you can find the field Control Plane IP from cluster detail page.

1. Get the namespace for Velero which is in the format of velero-<uuid>, for instance velero-5d440c9f-3fa7-4634-a1a3-2e0aa35110e7.

> kubectl get ns | grep velero

velero-5d440c9f-3fa7-4634-a1a3-2e0aa35110e7   Active   42m

2. Print the service account to console in yaml format.

> kubectl -n <your-velero-ns>  get sa ose-k8s-br -o yaml

apiVersion: v1 
kind: ServiceAccount 
metadata: 
  creationTimestamp: "2023-09-13T07:58:17Z" 
  name: ose-k8s-br 
  namespace: <your-velero-ns>
  resourceVersion: "656654" 
  uid: 5dbaba2b-eb31-4bf2-b10d-ed71d7bc365b

3. Create a file secret.yaml with below content. The ServiceAccount part should be got from step 2.

apiVersion: v1
kind: Secret
metadata:
  name: ose-k8s-br-secret
  namespace: <your-velero-ns>
  annotations:
    kubernetes.io/service-account.name: "ose-k8s-br"
type: kubernetes.io/service-account-token
---
<<insert-ServiceAccount-part-here>>
secrets:
- name: ose-k8s-br-secret

A full sample is like this:

apiVersion: v1
kind: Secret
metadata:
  name: ose-k8s-br-secret
  namespace: <your-velero-ns>
  annotations:
    kubernetes.io/service-account.name: "ose-k8s-br"
type: kubernetes.io/service-account-token
---
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  creationTimestamp: "2023-09-13T07:58:17Z" 
  name: ose-k8s-br 
  namespace: <your-velero-ns>
  resourceVersion: "656654" 
  uid: 5dbaba2b-eb31-4bf2-b10d-ed71d7bc365b
secrets:
- name: ose-k8s-br-secret

4. Apply the service account secret

# > kubectl apply -f secret.yaml

5. Try to protect the Kubernetes cluster from Object Storage Extension UI again.

Additional Information

Impact/Risks:
This issue affects to VMware Cloud Director Object Storage Extension 2.2.2 and earlier versions.