How to change Azure credentials in TKG
Article ID: 313129
Updated On:
Products
VMware Tanzu Kubernetes Grid
Issue/Introduction
This KB outlines the steps to change the Azure credentials.
Symptoms:
Credentials have expired and the capz-controller-manager is unable to authenticate with Azure. You will see the following error
E1105 15:20:07.339030 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile AzureMachine: failed to create network interface: failed to fetch network interface tkg-mgmt-az-md-0-k2qc4-nic: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/a1e178af-4a1a-4307-b438-09fb1acb2ddf/resourceGroups/tkg-ryanpe-rg/providers/Microsoft.Network/networkInterfaces/tkg-mgmt-az-md-0-k2qc4-nic?api-version=2019-06-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {\"error\":\"invalid_client\",\"error_description\":\"AADSTS7000215: Invalid client secret is provided.\\r\\nTrace ID: 227e3560-ac9f-4dcd-842f-c24c6e4d2700\\r\\nCorrelation ID: 30e3ee8a-9eba-4c34-a877-4d3d3e2b0283\\r\\nTimestamp: 2021-11-05 15:20:07Z\",\"error_codes\":[7000215],\"timestamp\":\"2021-11-05 15:20:07Z\",\"trace_id\":\"227e3560-ac9f-4dcd-842f-c24c6e4d2700\",\"correlation_id\":\"30e3ee8a-9eba-4c34-a877-4d3d3e2b0283\",\"error_uri\":\"https://login.microsoftonline.com/error?code=7000215\"}" "controller"="azuremachine" "name"="tkg-mgmt-az-md-0-k2qc4" "namespace"="tkg-system”
Symptoms:
Credentials have expired and the capz-controller-manager is unable to authenticate with Azure. You will see the following error
E1105 15:20:07.339030 1 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to reconcile AzureMachine: failed to create network interface: failed to fetch network interface tkg-mgmt-az-md-0-k2qc4-nic: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/a1e178af-4a1a-4307-b438-09fb1acb2ddf/resourceGroups/tkg-ryanpe-rg/providers/Microsoft.Network/networkInterfaces/tkg-mgmt-az-md-0-k2qc4-nic?api-version=2019-06-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {\"error\":\"invalid_client\",\"error_description\":\"AADSTS7000215: Invalid client secret is provided.\\r\\nTrace ID: 227e3560-ac9f-4dcd-842f-c24c6e4d2700\\r\\nCorrelation ID: 30e3ee8a-9eba-4c34-a877-4d3d3e2b0283\\r\\nTimestamp: 2021-11-05 15:20:07Z\",\"error_codes\":[7000215],\"timestamp\":\"2021-11-05 15:20:07Z\",\"trace_id\":\"227e3560-ac9f-4dcd-842f-c24c6e4d2700\",\"correlation_id\":\"30e3ee8a-9eba-4c34-a877-4d3d3e2b0283\",\"error_uri\":\"https://login.microsoftonline.com/error?code=7000215\"}" "controller"="azuremachine" "name"="tkg-mgmt-az-md-0-k2qc4" "namespace"="tkg-system”
Environment
VMware Tanzu Kubernetes Grid 1.x
VMware Tanzu Kubernetes Grid Plus 1.x
VMware Tanzu Kubernetes Grid Plus 1.x
Resolution
Retrieve the Azure credentials from Azure administrator and encode them
echo -n CREDENTIAL | base64 -w 0
Target the Management cluster and update the capz-manager-bootstrap-credentials secret
kubectl config use-context MGMT-CONTEXT
kubectl patch secret capz-manager-bootstrap-credentials -n capz-system --type merge -p '{"data":{"client-secret": "ENCODED-CREDENTIAL"}}'
Restart the Controller Manager pod
kubectl delete pod CONTROLLER_MANAGER-POD -n capz-system kubectl get pods -n capz-system
The azure json secrets will also be updated with new credentials
kubectl get secrets -A | grep azure-json
kubectl get secret AZURE-JSON-SECRET -o jsonpath='{.data.azure\.json}' | base64 -d
Azure CSI Drivers
If Azure CSI drivers have been installed on the cluster, please perform the following steps to rotate the credentials. When installing the Azure CSI Drivers, the administrator would have chosen to store the credentials in a secret on the cluster or else in a file on the Control Plane nodes.
Proceed with the corresponding steps below depending on which option was selected when the CSI driver were installed.
kubectl get secrets -n kube-system azure-cloud-provider -o jsonpath={'.data.cloud-config'} | base64 -d > cloud-config.yaml
vi cloud-config.yaml
kubectl create secret generic azure-cloud-provider --from-file=cloud-config=cloud-config.yaml -n kube-system -o yaml --dry-run | kubectl replace -f-
Restart the CSI Driver pods in kube-system namespace
kubectl get pods -n kube-system | grep csi
kubectl delete pod POD -n kube-system
The nodes will need to be redeployed to update azure.json.
For Control Plane nodes, add "upgradeAfter" to kubeadmControlPlane spec with a timestamp in RFC3339 UTC "Zulu" format
kubectl get kcp -A
kubectl edit kcp -n NAMESPACE KUBEADM-CONTROL-PLANE
spec:
replicas: 1
upgradeAfter: "2022-04-19T16:16:07Z"
For Worker nodes, delete the machine and it will be redeployed.
kubectl get machines -A
kubectl delete machine MACHINE-NAME -n namespace
echo -n CREDENTIAL | base64 -w 0
Target the Management cluster and update the capz-manager-bootstrap-credentials secret
kubectl config use-context MGMT-CONTEXT
kubectl patch secret capz-manager-bootstrap-credentials -n capz-system --type merge -p '{"data":{"client-secret": "ENCODED-CREDENTIAL"}}'
Restart the Controller Manager pod
kubectl delete pod CONTROLLER_MANAGER-POD -n capz-system kubectl get pods -n capz-system
The azure json secrets will also be updated with new credentials
kubectl get secrets -A | grep azure-json
kubectl get secret AZURE-JSON-SECRET -o jsonpath='{.data.azure\.json}' | base64 -d
Azure CSI Drivers
If Azure CSI drivers have been installed on the cluster, please perform the following steps to rotate the credentials. When installing the Azure CSI Drivers, the administrator would have chosen to store the credentials in a secret on the cluster or else in a file on the Control Plane nodes.
Proceed with the corresponding steps below depending on which option was selected when the CSI driver were installed.
- Credentials stored in Secret
kubectl get secrets -n kube-system azure-cloud-provider -o jsonpath={'.data.cloud-config'} | base64 -d > cloud-config.yaml
vi cloud-config.yaml
kubectl create secret generic azure-cloud-provider --from-file=cloud-config=cloud-config.yaml -n kube-system -o yaml --dry-run | kubectl replace -f-
Restart the CSI Driver pods in kube-system namespace
kubectl get pods -n kube-system | grep csi
kubectl delete pod POD -n kube-system
- Credentials stored in azure.json on Control Plane nodes
The nodes will need to be redeployed to update azure.json.
For Control Plane nodes, add "upgradeAfter" to kubeadmControlPlane spec with a timestamp in RFC3339 UTC "Zulu" format
kubectl get kcp -A
kubectl edit kcp -n NAMESPACE KUBEADM-CONTROL-PLANE
spec:
replicas: 1
upgradeAfter: "2022-04-19T16:16:07Z"
For Worker nodes, delete the machine and it will be redeployed.
kubectl get machines -A
kubectl delete machine MACHINE-NAME -n namespace
Feedback
Yes
No
Powered by
