Concourse does not recover after ATC disconnects from database

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Concourse does not recover after ATC disconnects from the database.

Symptoms

1. Experience turbulence error on the UI.

2. Not able to login because it is returning "Bad Request" 

3. External postgres database is used.

4. ATC web logs shows error like the one below:

fetch teams: write tcp <ATC web IP>:52952-><postgres database IP>:5432: write: connection reset by peer

Environment

Product Version: 2.10

Resolution

There is a bug in the postgres driver included in Concourse: https://github.com/concourse/concourse/issues/6726

The fix is to upgrade Concourse to 6.7.6 which contains an updated driver: https://github.com/concourse/concourse/releases/tag/v6.7.6

Bump lib/pq to 1.10.0 which fixes a regression in lib/pq where under certain circumstances the driver would not drop dead connections and never recover.