[Tanzu Application Service (TAS) Metrics v1] When metrics ingestor points error to Redis

Products

• Operations Manager

Issue/Introduction

Note: Pivotal Application Service (PAS) is now VMware Tanzu Application Service (TAS) for VMs.

In TAS Metrics v1, the frontend app metrics-ingestor receives app logs, container metrics, and network metrics from the Firehose and forwards them to the Redis job running in a dedicated VM. See TAS Metrics v1 architecture.

This article discusses what to do when you see one of the following errors from the metrics-ingestor logs:

1. Aggregation Stored Procedures key is not in redis

2. NOSCRIPT No matching script. Please use EVAL.
   

Note: In v2, TAS Metrics has been renamed App Metrics which has been rearchitectured and does not have the Redis component. See App Metrics v2 architecture. Therefore, the workaround in this article only applies to TAS Metrics v1.

Resolution

Check the recent log of the metrics-ingestor app.

​​​​​cf target -o system -s metrics-v1-6   ## the name of space depends on your Metrics version

cf logs --recent metrics-ingestor

You may see one of the following errors:

NewBuf ERROR: got an error from redis while loading the script: : <nil>
   2020-07-14T05:39:09.99-0500 [APP/PROC/WEB/1] ERR panic: Aggregation Stored Procedures key is not in redis

insert ERROR: Could not execute redis command `EVALSHA d6e840de2b624ff4076a149fc4efc405e7a9c551 0 2.122622442e+09 app_metric_9808f25b_9f23_4bb6_98e5_9fd1485518a3_1597891440 {"instanceKey":"9808f25b-9f23-4bb6-98e5-9fd1485518a3/1/1","name":"cf.system.disk.quota","subType":"avg","unit":"bytes"} {"instanceKey":"9808f25b-9f23-4bb6-98e5-9fd1485518a3/1/1","name":"cf.system.disk.quota","subType":"count","unit":"bytes"}`: NOSCRIPT No matching script. Please use EVAL.

The metrics-ingestor app (as a Redis client) needs to evaluate two Lua scripts in Redis, which are loaded into the script cache of Redis by the BOSH post-start script (/var/vcap/jobs/redis/bin/post-start) located in the Redis VM. When the Redis job is restarted, the loaded Lua scripts will be removed from script cache of Redis. If the Redis job restart is caused by some non-BOSH VM operations, the post-start script will not be triggered, thus the Lua scripts will not be reloaded into the script cache of Redis.

Example non-BOSH VM operations:

• "monit restart redis"

• Redis VM rebooting

When the Lua scripts do not exist in the script cache of Redis, the metrics-ingestor app will throw errors as mentioned above.

To resolve this issue, run the Redis job post-start script to reload the Lua scripts into Redis.

Run Redis job post-start script

1. Log into the director.

2. Run bosh instances and look for the Redis instance under the TAS Metrics v1 deployment (apmPostgres-<GUID>):

$ bosh instances
Deployment 'apmPostgres-xyz'

Instance                                            Process State  AZ   IPs
errand-runner/ae5a6140-5169-4354-b1c9-6e4e999eec92  running        az1  10.193.80.115
mysql/05e5dc49-15f6-4370-86ec-f08ef227a191          running        az1  10.193.80.113
postgres/c2491047-2c12-46fd-a76f-47a08762bbc6       running        az1  10.193.80.114
redis/c064aba8-022b-46f4-94d3-d6ee7a7dc7a2          running        az1  10.193.80.112

3. Run the command to SSH into redis VM: bosh -d apmPostgres-<GUID> ssh redis/<GUID>

$ bosh -d apmPostgres-xyz ssh redis/c064aba8-022b-46f4-94d3-d6ee7a7dc7a2
Using environment '1.1.1.1' as client 'ops_manager'
......
redis/c064aba8-022b-46f4-94d3-d6ee7a7dc7a2:~$

4. Run the following commands to execute the post-start script of the Redis job:

# sudo -i
# /var/vcap/jobs/redis/bin/post-start

5. The output would look like the following:

# ./post-start
+ PASSWORD=<redacted>
+ PORT=6379
+ REDIS_CLI=/var/vcap/packages/redis/bin/redis-cli
+ export 'REDIS_CMD=/var/vcap/packages/redis/bin/redis-cli -p 6379 -a <redacted> -n 1'
+ REDIS_CMD='/var/vcap/packages/redis/bin/redis-cli -p 6379 -a <redacted> -n 1'
+ wait_for_redis
+ tries=0
+ '[' 0 -lt 60 ']'
+ /var/vcap/packages/redis/bin/redis-cli -p 6379 -a <redacted> -n 1 info persistence
+ grep loading:0
loading:0
+ echo 'Redis is up and available.'
Redis is up and available.
+ return 0
+ /var/vcap/packages/metrics-data/src/github.com/pivotal-cf/metrics-data/redis/load_stored_procedures.sh
++ dirname /var/vcap/packages/metrics-data/src/github.com/pivotal-cf/metrics-data/redis/load_stored_procedures.sh
+ SCRIPT_DIR=/var/vcap/packages/metrics-data/src/github.com/pivotal-cf/metrics-data/redis
+++ cat /var/vcap/packages/metrics-data/src/github.com/pivotal-cf/metrics-data/redis/gauge_aggregation.lua
++ /var/vcap/packages/redis/bin/redis-cli -p 6379 -a <redacted> -n 1 SCRIPT LOAD '-- FIXME - WARNING: if you change these files you need to updated the hardcoded shas in the go
local value = ARGV[1]
local key = ARGV[2]
local avgField = ARGV[3]
local countField = ARGV[4]

local newCount = redis.call("HINCRBY", key, countField, 1)
if newCount == 1 then
    redis.call('\''HSET'\'', key, avgField, value)
else
    local currentAvg = redis.call("HGET", key, avgField)
    local newAvg = (value - currentAvg) / newCount + currentAvg
    redis.call('\''HSET'\'', key, avgField, newAvg)
end'
+ GAUGE_SHA=<redacted>
+++ cat /var/vcap/packages/metrics-data/src/github.com/pivotal-cf/metrics-data/redis/counter_aggregation.lua
++ /var/vcap/packages/redis/bin/redis-cli -p 6379 -a <redacted> -n 1 SCRIPT LOAD '-- FIXME - WARNING: if you change these files you need to updated the hardcoded shas in the go
local value = ARGV[1]
local key = ARGV[2]
local field = ARGV[3]

local currentValue = redis.call("HGET", key, field)
if not currentValue or value > currentValue then
    redis.call('\''HSET'\'', key, field, value)
end'
+ COUNTER_SHA=<redacted>
+ echo 'GAUGE: <redacted> COUNTER: <redacted>'
GAUGE: <redacted> COUNTER: <redacted>
redis/<redacted>-

8. Run the following to see if the script executed successfully:
Note: 0 == SUCCESS

# echo  $?
0

 

Restart metrics-ingestor App

1. Log into cf CLI.

2. Find the metrics space:

MacBook-Pro-3 dmathis$ cf spaces | grep metric
metrics-v1-5

3. Run the following command:

$ cf login -a api.run -u admin -p <redacted> -o system -s metrics-v1-5 --skip-ssl-validation
API endpoint: api.run
Authenticating...
OK

Targeted org system

Targeted space metrics-v1-5

API endpoint:   https://api.run (API version: 2.112.0)
User:           admin
Org:            system
Space:          metrics-v1-5

4. Restart the metrics-ingestor app:

$ cf restart metrics-ingestor
Restarting app metrics-ingestor in org system / space metrics-v1-5 as admin...

Stopping app...

Waiting for app to start...

name:              metrics-ingestor
requested state:   started
routes:            ingestor.run
last uploaded:     Fri 29 Mar 06:02:21 EDT 2019
stack:             cflinuxfs2
buildpacks:        binary

type:            web
instances:       1/1
memory usage:    512M
start command:   ./ingestor/ingestor
     state     since                  cpu    memory         disk          details
#0   running   2019-04-04T14:21:32Z   0.6%   9.4M of 512M   15.2M of 1G

5. Wait a few minutes and confirm that metrics are showing up in the UI.
Note: This could take up to about 10 minutes.

You can verify if the Lua scripts exist in Redis cache be doing the following:

1. Find the Lua scripts' SHA from the last line of post-start script executing log:

# tail -1 /var/vcap/sys/log/redis/post-start.stderr.log
+ echo 'GAUGE: d6e840de2b624ff4076a149fc4efc405e7a9c551 COUNTER: dd6ba34920c85706a0b4f23274fbc638a7272ba0'

2. Find the credential to connect to Redis:

# cat /var/vcap/sys/log/redis/post-start.stderr.log  | grep "SCRIPT LOAD"
++ /var/vcap/packages/redis/bin/redis-cli -p 6379 -a U6nElZMlp26d3UnHVGdDTl1MkpccOv -n 1 SCRIPT LOAD '......

3. Use SCRIPT EXIST to check if the Lua scripts exist or not:

# /var/vcap/packages/redis/bin/redis-cli -p 6379 -a U6nElZMlp26d3UnHVGdDTl1MkpccOv -n 1 SCRIPT EXISTS d6e840de2b624ff4076a149fc4efc405e7a9c551
1) (integer) 1     <<<   return value 1 means the script exists

# /var/vcap/packages/redis/bin/redis-cli -p 6379 -a U6nElZMlp26d3UnHVGdDTl1MkpccOv -n 1 SCRIPT EXISTS dd6ba34920c85706a0b4f23274fbc638a7272ba0
1) (integer) 1