Autoscaler Failing on SQL Query- "panic: gorp: multiple rows returned"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

Autoscaler is failing on SQL query, and you see the following go panic error:

2017-06-12T22:24:05.38+0000 [APP/PROC/WEB/0]ERR 
panic: gorp: multiple rows returned for: SELECT * FROM autoscale_instances WHERE is_leader=true - []

2017-06-12T22:24:05.38+0000 
[APP/PROC/WEB/0]ERR goroutine 55 [running]:

2017-06-12T22:24:05.38+0000 
[APP/PROC/WEB/0]ERR cf-autoscaling/db/repositories.(*AutoscaleInstanceRepository)
.FindLeader(0xc420115020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5b1417b02bc1e944, 0xcc86830f30ecaa08)

 

Environment

Cause

This error occurs because Autoscaler gets into an inconsistent state and can’t determine it’s leader. This is due to a race condition. This can also occur when upgrade to PCF 1.11 mentioned here App Autoscaler Panic. This is fixed in PCF 1.10.9 and later

 

Resolution

An operator can clear this by going into the ERT MySQL instance to clear out the leader and then reestablish the quorum by scaling down and restarting the Autoscaler app one instance at a time. Below are the steps for fixing the leader status in the ERT MySQL Database:

1. Update autoscale_instances SET is_leader=0 WHERE is_leader=1;

2. Restart App

3. Once started, scale up to 2 instances. Start app and wait until finished

4. Then scale up to 3 instances and start app

5. You can see the events summary indicating that the query has passed