Error: Received error - {"code":"internal_error","message":"db: key not found","child_messages":null}

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• Error while trying to delete the Management Cluster from TCA UI
• Unable to Delete Management Cluster in Telco Cloud Automation UI
• Failed to delete the management cluster.
• INTERNAL_SERVER_ERROR    Internal Server error    Received error - {"code":"internal_error","message":"db: key not found","child_messages":null}
  
  
  
  
  

Environment

TCA: 2.3

TCP: 3.0

Cause

The issue seems to be that KBS cannot find the tkgcontext query data in the KBS database, so the deletion failed.
To check this, follow the procedure below:

1. Find Management Cluster UUID from TCA-CP CLI
   curl http://localhost:8888/api/v1/managementclusters
   
   
2. Query MC spec on tca-cp
   curl --request GET http://127.0.0.1:8888/api/v1/managementcluster/{MC_ID} | jq
   
   
3. Get tkgID from the response step 2
   
   
4. Confirm if the TKG context exists there.
   curl --request GET http://127.0.0.1:8888/api/v1/tkgcontext/{TKG_ID} | jq


5. Missing tkgcontext should give the following output:
   {"code":"not_found","message":"tkgcontext: #########-####-#####-####-############ not exist in DB.","child_messages":null}

Resolution

1. Re-create a New tkgcontext.json using the content of /opt/vmware/k8s-bootstrapper/mgmt_{MC_NAME}.yaml
   You can use the below sample file for tkgcontext.json

Scenario 1: Airgap Environment:

Below is the sample file for tkgcontext.json

{   "airgap": {     "caCert": "encoded certificate",     "fqdn": "airgap.example.com"   },   "id": "82ba7200-246d-4b31-a559-886bb6d3e311",   "infrastructure": {     "type": "vsphere",     "version": "8.0.2"   },   "ipFamilies": "IPv4",   "vsphere": {     "clusterName": "mgmt-cluster",     "dataCenter": "dual-dc",     "dataStore": "vsanDatastore",     "ip": "dual-vcprime.ipv6.com",     "network": "/dual-dc/network/mgmt/dual-vds/management",     "resourcePool": "/dual-dc/host/mgmt-cluster/Resources/management",     "username": "[email protected]",     "password": "Your Password"     "vmFolder": "capv",     "vmTemplate": "caas-photon-5-kube-v1.28.4+vmware.1-23424098"   } }

Scenario 2: Internet Connected Environment:

Below is the sample file for tkgcontext.json

{"id":"61353bdc-5701-49cc-9ea7-0a3d06a4fca5", "ipFamilies":"IPv4", "infrastructure":{ "type":"vsphere", "version":"7.0.3"}, "vsphere":{"ip":"vcenter.vmlabs.com", "username":"[email protected]", "dataCenter":"Datacenter", "dataStore":"Data-Disk", "clusterName":"Cluster", "network":"/Datacenter/network/DSwitch-MGMT/TCA-NTW", "resourcePool":"/Datacenter/host/Cluster/Resources/TCA_Management", "vmFolder":"TCA_Machines", "vmTemplate":"/Datacenter/vm/TCA_Images/photon-3-kube-v1.24.10-vmware.1-tkg.1-fbb49d-21438772"}}

1. To add the information of the Management Cluster, please use the file below from TCA-Control Plane:
   /opt/vmware/k8s-bootstrapper/mgmt_{MC_NAME}.yaml
   
   
2. Save the file on the /tmp location of the TCA-CP and run the commands below as root:
   su root
cd /tmp
curl --request POST "http://127.0.0.1:8888/api/v1/tkgcontext" -d @tkgcontext.json
   
   
3. From the TCA GUI, retry "Force Delete" Management Cluster