DX APM - Removing service data and fixing partitioning

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

You need steps by steps instructions how to remove apm service together with its data to deploy a new service instance

 

Environment

APM 20x

Resolution

Skip to end of metadata

Go to start of metadata

There are certain steps on how to safely remove the old instance together with its data.

1. In Kubernetes/Openshift, navigate to deployments and scale down desired service instance (for example apmservices-tas-001) deployment to 0. Wait until all service instance pods (for example apmservices-tas-001-758b59546b-zrv8d) are deleted.

   1. If you do not have access to Kubernetes dashboard, you can use kubectl. Please make sure you are connected to the correct namespace and then use command:

      oc scale --replicas=0 deployment --all

2. Navigate to nfs storage and delete folders with its contents:
   nfs/dxi/apmservices/tas-001/data
   nfs/dxi/apmservices/tas-001/logs
3. Recreate the folders:
   nfs/dxi/apmservices/tas-001/data
   nfs/dxi/apmservices/tas-001/logs
4. Change ownership of new folders: chown 1010 /data /logs
5. Get the master token for your apmservices.
   1. Open Openshift/Kubernetes
   2. Select correct project/namespace where your services are deployed
   3. Navigate to secrets
   4. Open apmservices-private
      1. In Open shift: click on "Reveal Secret"
      2. In Kubernetes: click on the "eye" icon next to "bootstrap.properties"
   5. Copy value of master token from field: "apm.security.tenantServiceMasterToken"

                 If you do not have access to Kubernetes dashboard, you can use kubectl:

1. 1. Get available secrets using command:

      kubectl get secrets

   2. Get content of apmservices-private
      

      kubectl get secret apmservices-private -o yaml

       

   3. Copy content of bootstrap.properties

   4. Decode the content using base64

      echo {content of bootstrap.properties} | base64 --decode

       

   5. Copy value of master token from field: "apm.security.tenantServiceMasterToken"
      
      

2. Get serviceId and InstanceId 

   1. You can get this from services tab in DX Cluster manager UI
      
      
      

   2. Or you can get it  by calling REST API:

      Method: GET URL: {apmservices}/tenants/node/fetch HEADERS:     Authorization: Bearer {masterToken} – master token from setp 5       CURL example: curl --location --request GET 'http://apmservices-gateway.IPADDRESS.nip.io/tenants/node/fetch' \ --header 'Authorization: Bearer {masterToken}'

3. Locate desired service by serviceID and copy serviceId and InstanceId

4. Remove mapping for requested service by calling REST API
   

   Method: POST URL: {apmservices}/tenants/node/deregister HEADERS:     Authorization: Bearer {masterToken} – master token from setp 5     Content-Type: application/json     Accept: application/json BODY: {   "serviceId": "{srviceId}", -- service id from step 6   "instanceId": "{instanceId}" --instance id from step 6 (if you used DX Cluster manager UI it was called "id") }   CURL example: curl --location --request POST 'http://apmservices-gateway.IPADDRESS.nip.io/tenants/node/deregister' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer {masterToken} \ --data-raw '{   "serviceId": "tas",   "instanceId": "e411a384-5091-4f23-9b48-722f722cde67" }'