Generating log bundles is failing with a 404 error on the VCFA

Products

• VCF Automation

Issue/Introduction

Generate Log bundle of the VCFA  from the UI erroed out with the following Error :

Error Code: OnCreateLogBundleFailed
OnCreateLogBundleFailed
API failed with response: {"statusCode":404,"responseMessage":"Not Found","outputData":"404 page not found\n","token":null,"contentLength":19,"allHeaders":null}

Environment

VMware Cloud Foundation 9.0

Cause

Support bundle is an asynchronous task. When it is triggered, a task id is returned, so that LCM could monitor progress. Caused by an error , an incorrect task id was returned, causing LCM to report the error 404.

Resolution

     As a workaround generate the logbundle using the following ' breakglass ' method.

 

     Requirements:

• SSH password for nodes (We are using it as ${SSH_PASSWORD} below)
• IP of one of the node VMs (${NODE_IP} below)

     Steps:

• SSH into the node

   ssh vmware-system-user@${NODE_IP}
   sudo -su root  

• Get a token and the IP address of the cluster 

  export KUBECONFIG=/etc/kubernetes/admin.conf
  K8S_TOKEN=$(kubectl get secrets synthetic-checker-krp -n vmsp-platform -ojsonpath={.data.token} | base64 -d)
  PRIMARY_VIP=$(kubectl get gateway/vmsp-gateway -n istio-ingress -ojsonpath={.status.addresses[0].value})

• List available bundles

 curl -k -XPOST -H "Authorization: Bearer ${K8S_TOKEN}" "https://${PRIMARY_VIP}:30005/webhooks/vmsp-platform/supportbundle/list"

     Output will look like this:

{"statusCode":200,"output":"File list length: 2\n","running":false,"outputValues":{"bundles":[{"mtime":"xxx, xx xxx xxxx hh:mm:ss GMT","name":"supportbundles/core-bundle-xxxxxxxx-xxxxxxxxx-xxxxx.tar.gz","size":xxxxxxxxxx},{"mtime":"xxx, xx xxx xxxx xx:xx:xx GMT","name":"supportbundles/vcfa-bundle-xxxxxxxx-xxxxxxxxx-  xxxxx.tar.gz","size":xxxxxxxxxx}]},"id":"cqlrzx","completedAt":"xxxx-xx-xxxx:xx:xx"}

    You can download any of these bundles from your machine. Make sure to set SSH_PASSWORD and PRIMARY_VIP accordingly. Bundle name is in the "name" field in the output above, like this:

curl -X GET -k -v -u -O "vmware-system-user:${SSH_PASSWORD}" https://${PRIMARY_VIP}/supportbundles/core-bundle-xxxxxxxx-xxxxxxxxx-xxxxx.tar.gz

• If no bundles are available, or you need to generate a new one, trigger support bundle generation from the node VM.

 curl -k -XPOST -H "Authorization: Bearer ${K8S_TOKEN}" https://${PRIMARY_VIP}:30005/webhooks/vmsp-platform/supportbundle/generate -d '{"component": "vcfa"}'

     It will collect data for the last 2 days.
     Output of this command has an "id" field that will be used to query the status of the command, i.e. "#####" here:

{
  "running": true,
  "id": "#####",
  "statusURI": "/webhooks/vmsp-platform/supportbundle/generate/#####",
  "completedAt": null
}

• Query the status of this command until "running" field becomes "false". ${id} is from previous step.

id="#####"
curl -k -XGET -H "Authorization: Bearer ${K8S_TOKEN}" https://${PRIMARY_VIP}:30005/webhooks/vmsp-platform/supportbundle/generate/${id}

     Repeat this command until the output looks like

{
  "output": "<logs from the job>",
  "running": false,
  "outputValues": {
    "download_url": "https://<IP>/supportbundles/vcfa-bundle-xxxxxxxx-xxxxxxxxx.tar.gz",
    "expires_at": "xxxx-xx-xxxxxx:xx:xxx",
    "status": {
      "message": "Successful",
      "success": true
    }
  },
  "id": "#####",
  "completedAt": "xxxx-xx-xxxxx:xx:xxx"
}

    If the job takes longer than 80 minutes, it will fail. If that happens, retry the generation from previous step with payload '{"component": "vcfa", "duration": "1"}'

• Use "download_url" value from the previous step to download the support bundle. This doesn't need to be done on this node.

curl -X GET -k -v -u "vmware-system-user:${SSH_PASSWORD}" -O ${download_url}