Update the vCenter Certificate change in TCA

Products

VMware Telco Cloud Automation

Issue/Introduction

All cluster, nodepool, and Network Function (NF) operations require a secure trust to vCenter via a vSphere certificate thumbprint.

Below error is observed in TCA when there is a mismatch in the VC thumbprint:

 [VcSessionEstablishmentFailed] vmconfig status is Failed, nodeconfig status is Normal.vmconfig failed: plugin placementPlugin check failed: POST "/sdk": 503 Service Unavailable

After renewing/replacing vCenter Certificate, the following steps must be followed to restore functionality to TCA 2.X

Environment

2.x

Resolution

Update the vCenter thumbprint in TCA and TKG after a vCenter certificate has been updated.

Obtain the vSphere certificate thumbprint from the Prepare to Deploy Management Clusters to vSphere page.
SSH into each TCA-CP that is attached to the vCenter as the admin user (for minikube).
Run the following command and take note of the Custom Resource (CR) name and namespace:
```
kubectl get VCenterPrime -A 
```
Run the following command to edit the CR of the target vCenter environment:
```
kubectl edit VCenterPrime -n <namespace> <cr_name>
```
Change the thumbprint field to match the vSphere certificate thumbprint from step 1.
Synchronize the Cell Site Domain Data.
Note: Wait until the Parent domain's status changes to PROVISIONED before proceeding.
Synchronize the Cell Site Host Data.
Note: Only resync the host(s) that have failed to provision as resyncing all hosts is a time consuming process.
Note: Ensure the partial resync checkbox is NOT checked to enable a full resync of the cell site host.

For each management cluster:

SSH into the management cluster control plane VIP with capv
Run the following command and note the CR name and namespace:
```
kubectl get VCenterPrime –A 
```
Edit the CR for the target vCenter environment with the following command:
```
kubectl edit VCenterPrime -n <namespace> <cr_name>
```
Change the "thumbprint" field to match the correct thumbprint from step 1.

Ensure the correct vCenter is being updated by verifying the IP address field in the CR.

apiVersion: telco.vmware.com/v1alpha1 
kind: VCenterPrime 
metadata: 
  name: vcprime-mgmt-cluster01 
  namespace: tca-system 
spec: 
  server: 
    address: 10.#.#.#  ===> verify the correct vCenter 
    credentialRef: 
      kind: Secret 
      name: vcprime-mgmt-cluster01-secret 
      namespace: tca-system 
  subConfig: 
    datacenter: tcpscale-VMCCloudDC 
    thumbprint: AB.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.## ===> change here

To re-import the TLS thumbprint for TCA-M/TCA-CP (vCenter self-signed certificates only)

Login to TCA-M/TCA-CP manage page with tca-m/tca-cp:9443

Re-import the new vCenter TLS thumbprint by “Administration” -> Certificate -> Trusted CA Certificate -> Click the “Import” button

To update the TLS thumbprint for TCA TKG Management Clusters

If vCenter Certificate was changed for a stretch cluster environment:

SSH into management cluster control plane VIP with capv and update the {mgmt-cluster-name}-vsphere-cpi-addon secret in the management cluster context
```
kubectl get secret -A | grep cpi-addon 
```

Save original CPI vSphere configure to temporary file

kubectl get secret -n tkg-system {mgmt-cluster-name}-vsphere-cpi-addon -o jsonpath='{.data.vsphereconf-custom\.lib\.txt}' |base64 -d >/tmp/vsphereconf.txt

Update the CPI vSphere config with new thumbprint in the temporary file

Sample as following of CPI vsphere config:

[root@tca /home/admin]# vim /tmp/vsphereconf.txt  
((@def vsphere_conf(): -@) 
[Global] 
user = "administrator@vsphere.local" 
password = "Password" 
port = "443" 
datacenters = "test-dc, cellsite-dc" 
[VirtualCenter "10.x.x.x"] 
datacenters = "test-dc" 
thumbprint = "CD.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##" ===> change to new thumbprint 
ip-family = "ipv4" 
[VirtualCenter "sc2-10-185-10-130.example"] 
datacenters = "cellsite-dc" 
thumbprint = "EF.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##" 
ip-family = "ipv4" 
[Workspace] 
server = "10.x.x.x" 
datacenter = "test-dc" 
thumbprint = "CD.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##" ===> change to new thumbprint 
ip-family = "ipv4"

Encode the CPI vsphere configure with new thumbprint

export encoded_vsphereconf_content=`base64 -w 0 /tmp/vsphereconf.txt`

Update the secret {mgmt-cluster-name}-vsphere-cpi-addon in tkg-system namespace in the management cluster context, then wait for Kapp reconciliation. Once reconciled, vsphere-cloud-config configmap in kube-system namespace will be updated.
```
kubectl patch secret {mgmt-cluster-name}-vsphere-cpi-addon -n tkg-system -p '{"data": {"vsphereconf-custom.lib.txt":"'${encoded_vsphereconf_content}'"}}' 
```
Verify the configmap is updated using the below command on the management cluster context:
```
kubectl -n kube-system get cm vsphere-cloud-config -o yaml
```
Restart the vsphere-cloud-controller-manager pod so that the new configmap is mounted
```
kubectl rollout restart ds/vsphere-cloud-controller-manager -n kube-system 
```

Additional Information

vSphere TLS Thumbprint may also be included in the vspherecluster CR and in some cases must be updated in both management and all workload cluster contexts.

Verify if thumbrpints are being used with the following command:

kubectl get vsphereclusters -A | grep -v READY | awk '{print $1,$2}' | while read namespace cluster ; do kubectl get vsphereclusters -n $namespace $cluster -o yaml | grep thumbprint ; done

If vspherecluster CR needs to be updated, use the following steps :

SSH into management cluster control plane VIP with capv

List all the vsphereclusters including the management cluster:

kubectl get vsphereclusters -A

NAMESPACE    NAME                AGE 

default-ns   tkg-test-workload   62d 

default-ns   tkg-wld             83d 

tkg-system   tkg-mgmt-cluster    83d

For each cluster, edit the vspherecluster using the below command and update the "spec.thumbprint" field with the correct thumbprint:
```
kubectl edit vsphereclusters -n default-ns tkg-test-workload 
```

Verify if the update is completed using the below command:

kubectl get vsphereclusters -n default-ns tkg-test-workload -o yaml

Note: For management clusters, use the tkg-system namespace to the kubectl commands:

kubectl edit vsphereclusters -n tkg-system tkg-mgmt-cluster