How to see the remaining amount of memory on Diego Cells in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

This article discusses how to determine the amount of remaining memory on Diego Cells in all versions of Tanzu Application Service for VMs (TAS for VMs).

Determining the remaining amount of memory on Diego Cells helps you determine whether there are sufficient resources for pushed applications.

When you run the command, cf push <APP> it returns the message "Insufficient Resources". You may need to perform this command to check whether there is enough remaining memory for the application that is being pushed.

Environment

Product Version: 2.2

Resolution

To find out the remaining amount of memory on a Diego Cell, follow the instructions below:

Using cfdot

You can manually pull capacity stats from Diego using the cfdot command. While this approach is works for short term troubleshooting, we do not suggest doing this for long term metric capture or monitoring solutions. Those solutions should use the firehose.

To pull the stats with cfdot, run the following command from the BBS, Brain, or from a Diego Cell:

cfdot cell-states | jq '{cell_id, TotalResources, AvailableResources}'

This will return a JSON blob similar to the one below:

{
  "cell_id": "4c24989c-4cfb-41c9-a9c6-5b07b0e67eb4",
  "TotalResources": {
    "MemoryMB": 16383,
    "DiskMB": 121518,
    "Containers": 74
  },
  "AvailableResources": {
    "MemoryMB": 13311,
    "DiskMB": 118446,
    "Containers": 71
  }
}
...

There are other attributes available from the command above, such as the application instances running on each Diego Cell. If you'd like to see them adjust the filter specified to jq.

If querying a windows Diego Cell, the following can be used from within Linux Diego VMs, such as BBS or Brain.

cfdot cell-states | jq -c 'select( .cell_id == "windows cell id" ) | {cell_id, TotalResources, AvailableResources}' | jq

Using curl

To pull the stats with curl, run the following on your Cell:

curl -s https://localhost:1801/state --cert /var/vcap/jobs/rep/config/certs/tls.crt --key /var/vcap/jobs/rep/config/certs/tls.key --cacert /var/vcap/jobs/rep/config/certs/tls_ca.crt | jq '{cell_id, AvailableResources, TotalResources}'

This will return a JSON blob that looks like this:

{
  "cell_id": "202cc6c3-f950-4205-a3ae-6d9d6a325e9d",
  "AvailableResources": {
    "MemoryMB": 7466,
    "DiskMB": 174247,
    "Containers": 208
  },
  "TotalResources": {
    "MemoryMB": 32174,
    "DiskMB": 216687,
    "Containers": 249
  }
}

There are other attributes available from the command above, such as the application instances running on each Cell. If you'd like to see them, adjust the specified filter to jq.

Using the firehose

If you are building a long term monitoring solution, you will need to pull metrics from the firehose. For more information on monitoring capacity through the firehose, refer to Diego Cell Metrics.

It can also be helpful to use the cf firehose plugin for the CF CLI to view metrics. For more information on the cf firehouse plugin, refer to Installing the Loggregator Firehose Plugin for cf CLI. This lets you connect to the firehose and stream metrics. You can then filter those to pick out specific data points.

Keep in mind that metrics are only emitted to the firehose on a periodic basis. If you're not seeing a metric, make sure that you've been monitoring for at least five minutes to give sufficient time for a metric to be emitted and visible through the firehouse.

Older versions of TAS for VMs had an unsecured endpoint for the Diego Cell, with the same output. You can access this with the following curl command:

curl -s -k https://localhost:1800/state

Additional Information

Older versions of PCF had an unsecured endpoint for the Diego Cell, with the same output. You can access this with the following curl command:
 

curl -s -k https://localhost:1800/state