Concourse database full

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Concourse deployments have been failing with "Expected to find variables: git_private_key" upon further investigation it look like the concourse deployment database db/########-####-####-####-############, is full. An attempt to clean up the database results in "No space left on device" error.

 

db/########-####-####-####-############:~# df -h 
Filesystem Size Used Avail Use% Mounted on 
/dev 3.9G 4.0K 3.9G 1% 
/dev tmpfs 799M 300K 799M 1%
/run /dev/vda1 2.8G 1.2G 1.5G 44% 
/dev/vdb1 962G 913G 0 100%
/var/vcap/store

 

### relation                       | total_size
-----------------------------------+------------
 public.request_audit_record       | 631 GB
 public.event_audit_record         | 278 GB
 public.auth_failure_audit_record  | 1560 MB
 public.credential                 | 400 kB
 public.access_entry               | 296 kB
 public.credential_name            | 296 kB
 public.schema_version             | 80 kB
 public.encryption_key_canary      | 64 kB
 public.password_secret            | 16 kB
 public.user_credential            | 16 kB
 public.certificate_credential     | 16 kB
 public.ssh_credential             | 16 kB
 public.rsa_credential             | 16 kB
 public.hibernate_sequence         | 8192 bytes
(14 rows)

db/########-####-####-####-############:~# /var/vcap/packages/postgres-9.6.6/bin/psql -U credhub 
psql (9.6.6)
Type "help" for help.

credhub=> VACUUM FULL;
ERROR:  could not access status of transaction 0
DETAIL:  Could not write to file "pg_subtrans/1BF0" at offset 114688: No space left on device.
credhub=>

Environment

Concourse

Resolution

 

To free up space, you must enlarge the persistent disk. This requires creating a new, larger disk type and assigning it to the persistent disk in the Concourse manifest.

1. Create a new disk type

   • Run bosh cloud-config and capture the output into a file.

   • Edit the file to add a new persistent_disk_type definition.

   • Apply the changes by running:

      

     bosh update-cloud-config <edited-file>.yml

2. Update the Concourse deployment manifest

   • Run bosh -d concourse manifest > manifest.yml to capture the current manifest.

   • Edit the manifest so that the db instance uses the new persistent_disk_type for its persistent disk.

   • Redeploy the changes by running:

      

     bosh -d concourse deploy manifest.yml

After redeployment, the db instance uses the larger persistent disk.