How to get into an App Container manually with Garden-RunC backend

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Due to security reasons, some customers do not allow SSH access to app containers via the `cf ssh` command. At the same time, it can sometimes be useful to enter into a app's containers for troubleshooting purposes. This articles covers a backdoor method for getting into a container on a Diego cell.

Resolution

This procedure outlines the steps to manually enter into an app container where Garden-runC backend is running.

Note: Garden-runC backend is available only on Elastic Runtime (ERT) versions below 1.65, below 1.7.28, below 1.8.9, and below 1.9.0. Previous versions of ERT are shipped with the Garden-Linux backend. 

Method using cfdot utility

 

1. Establish the process-guid for the app:

 cf curl /v2/apps/$(cf app <app_name> --guid)| jq -r '.metadata.guid + "-" + .entity.version'
b06b2901-2093-4be4-9f37-f7788d04ce8c-ff93f0e2-285f-4f18-a6fc-806802f3dde9

2. Find the hosts where the app instances are running:

cf curl v2/apps/$(cf app <app-name> --guid)/stats | grep host
         "host": "10.193.73.14",
         "host": "10.193.73.13",

3. Identify the Diego cell you want to ssh to by running bosh vms | grep <cell_ip>

4. bosh ssh to the Diego cell and grab the cell_id from step 3.

5. Once in the Diego cell, as root, run the following command to grab the container_guid/instance_guid:

cfdot actual-lrps --process-guid <process_guid_from_step_1> | jq '.instance | select (.address == "<diego_cell_ip>")' | jq .instance_guid

6. Run the following command to get access to the container:

sudo /var/vcap/packages/runc/bin/runc --root /run/containerd/runc/garden exec -t <instance_guid> /bin/bash

Alternate method

1. Find the app GUID for the app container that we want to enter into using cf app <app-name> --guid, where <app-name> is the name of the app.

For example:

$ cf app spring-music --guid
e0afa9f2-4020-4da1-919c-59d1c66d7d3c  

2. Find the IP of the Diego Cell where the container app is hosted:

cf curl /v2/apps/<app-guid>/stats | grep -w host

Where <app-guid> is the GUID of the app from the command in step 1.

For example:

cf curl /v2/apps/e0afa9f2-4020-4da1-919c-59d1c66d7d3c/stats | grep -w host
       "host": "10.193.76.23", 

Note: There may be multiple IPs returned from the above command, similar to when there are multiple app instances. In most cases, app instances will be split across multiple Cells so each one will have its own IP address. Pick one of the IP addresses to use in the next step and we'll enter the app instance container running on that Cell.

3. From the Ops Manager VM, locate the name of your deployment using `bosh deployments`. It will be in the form `cf-<uuid>` for Elastic Runtime or Tanzu Application Service. Run `export BOSH_DEPLOYMENT=<deployment-name>`.

4. Now find the name of the Diego Cell from the IP using the `bosh instances | grep <IP>` command, where IP is the IP address from step 2.

$ bosh instances | grep 10.193.82.13
diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f                    	            running	main	10.193.82.13

The VM name is denoted by "<job_name>/<uuid>". From the above command, "diego_cell" is the job name and index is "0", you would see `compute/<uuid>` if you're using the Small Footprint variant of TAS for VMs.

5. SSH to Diego cell using `bosh ssh <job_name>/<uuid>` command. Replace <name> and <uuid> with the the values from step 3. You can then run `sudo -i` to become root.

For example:

$ bosh ssh diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f
Using environment '10.193.82.11' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Using deployment 'cf-435c6aa7c377367d89f3'

Task 7625. Done
Unauthorized use is strictly prohibited. All access and activity
...

diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f:~$ sudo -i
diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f:~#

6. Once you SSH into the Diego Cell, we can obtain the <container-guid> of the app by running the following command:

grep rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-run.node-run.monitor-run.run-step.running \
/var/vcap/sys/log/rep/rep.stdout.log | grep <app-guid> | tail -1 | python -m json.tool | grep 'container-guid'

Replace <app-guid> in the above command with the app-guid of the app from step 1.

For example:

# grep rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-run.node-run.monitor-run.run-step.running \
/var/vcap/sys/log/rep/rep.stdout.log | grep e0afa9f2-4020-4da1-919c-59d1c66d7d3c | tail -1 | python -m json.tool | grep 'container-guid'
 "container-guid": "891090cc-259f-4648-4d45-43ea0d8d2af9"

7. If you're using one of the following versions, you can skip steps 7 and 8, and proceed with step 9.

Elastic Runtime:

• ERT >= 1.8.40
• ERT >= 1.9.18
• ERT >= 1.10.5
• ERT 1.11 (all)
• ERT 1.12 (all)

Using the value of container-guid from output of the command in step 6, change the directory to the /var/vcap/data/garden/depot/<container-guid> directory. From the output in step 6, the container-GUID is 891090cc-259f-4648-4d45-43ea0d8d2af9. 

# cd  /var/vcap/data/garden/depot/891090cc-259f-4648-4d45-43ea0d8d2af9

8. Enter the guardian mount namespace:

# /var/vcap/packages/guardian/bin/inspector-garden -pid $(pidof guardian) /bin/bash

9. Drop into the container shell as root by running the following command:

# /var/vcap/packages/runc/bin/runc exec -t <container-guid> /bin/bash

Alternatively, to enter the container as a non-root user such as vcap, run the following command:

# /var/vcap/packages/runc/bin/runc exec -u 2000:2000 -t <container-guid> /bin/bash

10. To confirm you're in the container, you can check the app processes by running `ps -ef`.

# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 2016 ? 00:00:00 /proc/self/exe init
vcap 13 0 0 2016 ? 00:00:00 /tmp/lifecycle/diego-sshd --allowedKeyExchanges= --address=0.0.0.0:2222 --allowUnauthenticatedClients=false --inheritDaemonEnv=true --allowedCiphers= --allo
vcap 19 0 0 2016 ? 00:22:36 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.util.logging.config.file=/home/vcap/app/.java-buildpack/tomcat/conf/logging.properties -Djava.ut
root 8581 0 0 14:15 ? 00:00:00 /bin/bash
root 8606 8581 0 14:15 ? 00:00:00 ps -ef
root 20334 1 0 Jan07 ? 00:00:00 [sudo] <defunct>
root@891090cc-259f-4648-4d45-43ea0d8d2af9:/#  

 

The commands above assume that you're using v2 of the BOSH CLI. In Ops Manager 1.11 and Ops Manager 1.12, the name of the binary is `bosh2`. It switches to `bosh` in Ops Manager 2.0.

To SSH into a container with a garden-Linux backend, please follow How to login to an app container as a root. This article applies to the following Elastic Runtime versions:

• ERT 1.6.45 and above
• ERT 1.7.28 and above
• ERT 1.8.8 and above
• ERT 1.8.40 and above
• ERT Below 1.9.18
• ERT Below 1.10.5

 

Additional Information

The commands above assume that you're using v2 of the bosh cli. In Ops Manager 1.11 and 1.12, the name of the binary is `bosh2`. It switches to `bosh` in Ops Manager 2.0.

For SSH'ing into a container with garden-Linux backend, please follow the procedure here. 

This article applies to Elastic Runtime versions 

• 1.6.45 and above
• 1.7.28 and above
• 1.8.8 and above
• 1.8.40 and above
• Below 1.9.18
• Below 1.10.5