How to change Azure credentials in TKG

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

This KB outlines the steps to change the Azure credentials.

Symptoms:
Credentials have expired and the capz-controller-manager is unable to authenticate with Azure. You will see the following error

E1105 15:20:07.339030 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile AzureMachine: failed to create network interface: failed to fetch network interface tkg-mgmt-az-md-0-k2qc4-nic: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/a1e178af-4a1a-4307-b438-09fb1acb2ddf/resourceGroups/tkg-mgmt-az/providers/Microsoft.Network/networkInterfaces/tkg-mgmt-az-md-0-k2qc4-nic?api-version=2019-06-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {\"error\":\"invalid_client\",\"error_description\":\"AADSTS7000215: Invalid client secret is provided.\\r\\nTrace ID: 227e3560-ac9f-4dcd-842f-c24c6e4d2700\\r\\nCorrelation ID: 30e3ee8a-9eba-4c34-a877-4d3d3e2b0283\\r\\nTimestamp: 2021-11-05 15:20:07Z\",\"error_codes\":[7000215],\"timestamp\":\"2021-11-05 15:20:07Z\",\"trace_id\":\"227e3560-ac9f-4dcd-842f-c24c6e4d2700\",\"correlation_id\":\"30e3ee8a-9eba-4c34-a877-4d3d3e2b0283\",\"error_uri\":\"https://login.microsoftonline.com/error?code=7000215\"}" "controller"="azuremachine" "name"="tkg-mgmt-az-md-0-k2qc4" "namespace"="tkg-system”

Environment

VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid Plus 1.x

Resolution

Retrieve the Azure credentials from Azure administrator and encode them

echo -n CREDENTIAL | base64 -w 0

Target the Management cluster and update the capz-manager-bootstrap-credentials secret 

kubectl config use-context MGMT-CONTEXT
kubectl patch secret capz-manager-bootstrap-credentials -n capz-system --type merge -p '{"data":{"client-secret": "ENCODED-CREDENTIAL"}}'

Restart the Controller Manager pod
kubectl delete pod CONTROLLER_MANAGER-POD -n capz-system kubectl get pods -n capz-system

The azure json secrets will also be updated with new credentials
kubectl get secrets -A | grep azure-json
kubectl get secret AZURE-JSON-SECRET -o jsonpath='{.data.azure\.json}' | base64 -d


Azure CSI Drivers
If Azure CSI drivers have been installed on the cluster, please perform the following steps to rotate the credentials. When installing the Azure CSI Drivers, the administrator would have chosen to store the credentials in a secret on the cluster or else in a file on the Control Plane nodes.
Proceed with the corresponding steps below depending on which option was selected when the CSI driver were installed.

• Credentials stored in Secret

kubectl get secret azure-cloud-provider -n kube-system
kubectl get secrets -n kube-system azure-cloud-provider -o jsonpath={'.data.cloud-config'} | base64 -d > cloud-config.yaml
vi cloud-config.yaml
kubectl create secret generic azure-cloud-provider --from-file=cloud-config=cloud-config.yaml -n kube-system -o yaml --dry-run | kubectl replace -f-

Restart the CSI Driver pods in kube-system namespace
kubectl get pods -n kube-system | grep csi
kubectl delete pod POD -n kube-system

 

• Credentials stored in azure.json on Control Plane nodes

The /etc/kubernetes/azure.json file is created during initialisation of the node using data from secret MACHINE-NAME-azure-json.
The nodes will need to be redeployed to update azure.json.

For Control Plane nodes, add "upgradeAfter" to kubeadmControlPlane spec with a timestamp in RFC3339 UTC "Zulu" format
kubectl get kcp -A
kubectl edit kcp -n NAMESPACE KUBEADM-CONTROL-PLANE
spec:
  replicas: 1
  upgradeAfter: "2022-04-19T16:16:07Z"

For Worker nodes, delete the machine and it will be redeployed.
kubectl get machines -A
kubectl delete machine MACHINE-NAME -n namespace