DX APM - Removing service data and fixing partitioning

book

Article ID: 144011

calendar_today

Updated On:

Products

CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

You need steps by steps instructions how to remove apm service together with its data to deploy a new service instance

 

Environment

APM 11.x, 19.x

Resolution

Skip to end of metadata

 

Go to start of metadata

 

There are certain steps on how to safely remove the old instance together with its data.

  1. In Kubernetes/Openshift, navigate to deployments and scale down desired service instance (for example apmservices-tas-001) deployment to 0. Wait until all service instance pods (for example apmservices-tas-001-758b59546b-zrv8d) are deleted.
    1. If you do not have access to Kubernetes dashboard, you can use kubectl. Please make sure you are connected to the correct namespace and then use command:

      oc scale --replicas=0 deployment --all
  2. Navigate to nfs storage and delete folders with its contents:
    nfs/dxi/apmservices/tas-001/data
    nfs/dxi/apmservices/tas-001/logs
  3. Recreate the folders:
    nfs/dxi/apmservices/tas-001/data
    nfs/dxi/apmservices/tas-001/logs
  4. Change ownership of new folders: chown 1010 /data /logs
  5. Get the master token for your apmservices.
    1. Open Openshift/Kubernetes
    2. Select correct project/namespace where your services are deployed
    3. Navigate to secrets
    4. Open apmservices-private
      1. In Open shift: click on "Reveal Secret"
      2. In Kubernetes: click on the "eye" icon next to "bootstrap.properties"
    5. Copy value of master token from field: "apm.security.tenantServiceMasterToken"

                 If you do not have access to Kubernetes dashboard, you can use kubectl:

    1. Get available secrets using command:

      kubectl get secrets
    2. Get content of apmservices-private

      kubectl get secret apmservices-private -o yaml

       

    3. Copy content of bootstrap.properties
    4. Decode the content using base64

      echo {content of bootstrap.properties} | base64 --decode

       

    5. Copy value of master token from field: "apm.security.tenantServiceMasterToken"

  1. Get serviceId and InstanceId 

    1. You can get this from services tab in DX Cluster manager UI:


    2. Or you can get it  by calling REST API:

      Method: GET
      URL: {apmservices}/tenants/node/fetch
      HEADERS:
          Authorization: Bearer {masterToken} – master token from setp 5
           
      CURL example:
      --header 'Authorization: Bearer {masterToken}'
  2. Locate desired service by serviceID and copy serviceId and InstanceId
  3. Remove mapping for requested service by calling REST API

    Method: POST
    URL: {apmservices}/tenants/node/deregister
    HEADERS:
        Authorization: Bearer {masterToken} – master token from setp 5
        Content-Type: application/json
        Accept: application/json
    BODY:
    {
      "serviceId""{srviceId}", -- service id from step 6
      "instanceId""{instanceId}" --instance id from step 6 (if you used DX Cluster manager UI it was called "id")
    }
     
    CURL example:
    --header 'Accept: application/json' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: Bearer {masterToken} \
    --data-raw '{
      "serviceId""tas",
      "instanceId""e411a384-5091-4f23-9b48-722f722cde67"
    }'