App Metrics 2.0.0 is failing because it cannot connect to the Metric Store in environments using application security groups
Article ID: 293628
Updated On:
Products
Issue/Introduction
"Apply Changes" fails with the following error message:
Task 5668597 | 16:14:50 | Error: 'metric-store/47b1e7a4-####-####-####-ccf0e6c4ceef (0)' is not running after update. Review logs for failed jobs: route_registrar.
The appmetrics application deployment fails with log output similar to the following:
2020-04-16T15:30:29.200-05:00 [APP/PROC/WEB/0] [OUT] [1m2020-04-16 20:30:29[m [32mINFO[m [2mcatalyst.MetricsApplication[m [main] Running flyway migrations
2020-04-16T15:31:27.530-05:00 [HEALTH/0] [ERR] Failed to make TCP connection to port 8080: connection refused
2020-04-16T15:31:27.530-05:00 [CELL/0] [ERR] Failed after 1m0.632s: readiness health check never passed.
2020-04-16T15:31:27.902-05:00 [APP/PROC/WEB/0] [OUT] Exit status 143
2020-04-16T15:31:33.350-05:00 [CELL/0] [OUT] Cell 540f9b84-####-####-####-f79e8521a1fb stopping instance 3846cc4a-####-####-5a1d-7efd
2020-04-16T15:31:33.350-05:00 [CELL/0] [OUT] Cell 540f9b84-####-####-####-f79e8521a1fb destroying container for instance 3846cc4a-####-####-####-7efd
2020-04-16T15:31:33.375-05:00 [API/4] [OUT] Process has crashed with type: "web"
2020-04-16T15:31:33.401-05:00 [API/4] [OUT] App instance exited with guid a37bfca6-5183-4b0f-9099-4dd7418bb3bb payload: {"instance"=>"3846cc4a-8d90-4a9b-5a1d-7efd", "index"=>0, "cell_id"=>"540f9b84-####-####-####-f79e8521a1fb", "reason"=>"CRASHED", "exit_description"=>"Instance never healthy after 1m0.632s: Failed to make TCP connection to port 8080: connection refused", "crash_count"=>4, "crash_timestamp"=>1587069093334827777, "version"=>"7146978a-####-####-####-7de618dc3155"}
2020-04-16T15:31:33.525-05:00 [PROXY/0] [OUT] Exit status 137
2020-04-16T15:31:34.588-05:00 [CELL/0] [OUT] Cell 540f9b84-####-####-####-f79e8521a1fb successfully destroyed container for instance 3846cc4a-####-####-####-7efd
The application deployment takes a few minutes to fail depending on your foundation settings. The error message `Failed to make TCP connection to port 8080: connection refused` and `readiness health check never passed.` should be present in the log output.
Cause
The appmetrics application is failing to start because it is not able to connect to the database. As the result the database migration hangs and the platform restarts the container because the application never got healthy (never accepted connections).
Environment
Resolution
App Metrics 2.0.0 is not creating the required application security group.
1. Identify the IP address of the App Metrics database virtual machine (VM).
bosh deployments | grep -i appMetrics
b. Get the IP address:
$ bosh -d appMetrics-862b0####a650872f77a vms Using environment '10.###.##.11' as client 'ops_manager' Task 96734. Done Deployment 'appMetrics-862b0####a650872f77a' Instance Process State AZ IPs VM CID VM Type Active db-and-errand-runner/25ecaa10-####-####-a52d-c41e30d24d4a running az1 10.###.##.80 vm-f0dc20f5-64a4-4b9d-ac39-0991bd37b095 xlarge.cpu true log-store-vms/22251307-####-4c75-a848-fe785e6e7e43 running az2 10.###.##.94 vm-d0ac40e5-061a-4ae8-80be-e6086a80aada xlarge true log-store-vms/4576c20f-####-4a69-84f2-ada17b0ba116 running az1 10.###.##.93 vm-ae8bc5e9-e95b-490c-8c73-03c6c1ef7e70 xlarge true log-store-vms/6d8596b4-####-42cd-8c1e-284c0fc2a3db running az3 10.###.##.95 vm-46c5d296-ccff-4133-8618-f6d9a7eb2426 xlarge true 4 vms Succeeded
Take note of the IP address of the db-and-errand-runner virtual machine (VM).
2. Login to your foundation with an admin user:
cf login -a <api>
3. Target the system organization and app-metrics-v2 space:
cf target -o system -s app-metrics-v2
4. Add a security group to the foundation:
[
{
"protocol": "tcp",
"destination": "<IP-ADDRESS-FROM-STEP-1>/32",
"ports": "5432",
"description": "Allow traffic to app metrics database"
}
]
b. Create the application security group:
cf create-security-group app-metrics-database asg.json
5. Bind the application security group to App Metrics space:
cf bind-security-group app-metrics-database system app-metrics-v2
6. Rerun "Apply Changes" by do the following:
a. Navigate to the Ops Manager dashboard.
b. Click on the "Review Pending Changes" button.
c. Select the "App Metrics" product.
d. Click "Apply changes"
The "Apply changes" should complete successfully.
7. Ingest the indicators manually.
a. SSH into the db-and-errand-runner virtual machine in appmetrics deployment:
bosh ssh -d <app-metrics-deployment> db-and-errand-runner/0
b. Ingest the indicators manually:
export APP_NAME='appmetrics'
export SYSTEM_DOMAIN='<SYSTEM_DOMAIN>'
cd /var/vcap/jobs/push-app-metrics
curl https://${APP_NAME}.${SYSTEM_DOMAIN}/indicator-documents -H "Authorization: $(cf oauth-token)" --data-binary @./config/indicators.yml
Note: At this point, your metrics installation should be operational.
Feedback
