How to troubleshoot app which fails during start phase on TAS

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When an app passes staging successfully but fails to starts and only generates limited logs, we can simulate the app start with exact same container environment with start options manually for troubleshooting purpose.

Resolution

Scenario 1 - cf ssh is enabled

1. Find necessary information of the app droplet.

   • $ cf curl /v3/apps/$(cf app myapp --guid)/droplets/current
       "buildpacks": [
         {
           "name": "nodejs_buildpack",
           "detect_output": "nodejs",
           "buildpack_name": "nodejs",
           "version": "1.7.69"
         }
       ],
       "stack": "cflinuxfs3",
     ...
       "process_types": {
         "web": "npm start"
       },
     ...
       "metadata": {
         "labels": {
     
         },
         "annotations": {
     
         }
       },

2. Push the app with `sleep 3600` and process healthcheck to replace original start command and healthcheck. Please keeps other configuration as same as possible. Here is a nodejs sample.

   • $ cf push my-app -p ./myapp/ -c "sleep 3600" -u process -b nodejs_buildpack

3. SSH into the app container

   • $ cf ssh my-app
     vcap@34a2a79e-461a-472d-67ac-63b5:~$ 

4. Start app from buildpack lifecycle launcher to have the correct system context (working directory, environment variables, etc.).

   • vcap@34a2a79e-461a-472d-67ac-63b5:~$ /tmp/lifecycle/launcher /home/vcap/app bash ''
     vcap@34a2a79e-461a-472d-67ac-63b5:~$ npm start
     > start
     > node lib/index.js
     listening on port  8080

     or in single step

   • vcap@34a2a79e-461a-472d-67ac-63b5:~$ /tmp/lifecycle/launcher /home/vcap/app 'npm start' ''
     > start
     > node lib/index.js
     listening on port  8080

5. If above start fails, you may debug it with language specific steps, for example with npm you can enable debug flags

   • $ export NODE_DEBUG='timer,http,net,fs,cluster,tls,stream,child_process,module'
     $ /tmp/lifecycle/launcher /home/vcap/app 'npm start' ''
     ...
     MODULE 296: load "/home/vcap/app/lib/notifications_api.js" for module "/home/vcap/app/lib/notifications_api.js"
     MODULE 296: Module._load REQUEST ./fetch_helper parent: /home/vcap/app/lib/notifications_api.js
     MODULE 296: Module._load REQUEST ./success_api parent: /home/vcap/app/lib/invitations.js
     ...
     

     or with strace

   • $ strace -T -tt -f /tmp/lifecycle/launcher /home/vcap/app 'npm start' ''

Scenario 2 - cf ssh is not enabled
In this case,  please login to container as root  on diego cell instead. Then follow similar steps as below

$ su vcap
$ /tmp/lifecycle/launcher /home/vcap/app/ bash ''