BBR backup fails with: "Error streaming backup from remote instance. Error: ssh.Stream failed: stdout.Write failed: write... no space left on device".

Products

• VMware Tanzu Platform
• VMware Tanzu Application Service
• Concourse for VMware Tanzu
• Operations Manager

Issue/Introduction

When attempting to run a BBR backup of a TCPF deployment through concourse, you face an error like so:

Error streaming backup from remote instance. Error: ssh.Stream failed: stdout.Write failed: write <deployment/timestamp>/backup_restore-0-bbr-<component>.tar: no space left on device: ssh.Stream failed: stdout.Write failed

This leads to BBR failing.

Environment

TPCF environments.

Cause

This is due to a lack of ephemeral disk on the concourse worker to handle the backup of the tar file, and the extraction on the worker.

You can verify that the ephemeral disk is full by running df -h on the concourse workers, or bosh -d <concourse> vms --vitals | grep worker

Resolution

You will need to increase the concourse worker VMs to have a larger ephemeral disk. For example, to increase to 2xlarge.disk:

 

1. bosh -d <concourse_deployment> manifest > concourse.yml
2. vim concourse.yml
3. Search for vm_type under the worker section, and replace the current vm_type with a larger disk.
4. Redeploy with bosh -d <concourse_deployment> deploy concourse.yml

Then re-run the backup and confirm the disk is now large enough to handle the backup.