Concourse pipeline fails to run with 'failed to stream in to volume' error

Products

• Concourse for VMware Tanzu

Issue/Introduction

When running a Concourse pipeline, the pipeline fails with the following error message:

'failed to stream in to volume' 

This issue could be caused by a number of factors including the following:


1. The Concourse workers are stalled.

• The TSA component on the Web VM pings workers for a heartbeat every 30 seconds, updates ATC with worker metric changes, and if there is no response, notifies ATC that the worker is in a stalled state. A stalled worker is a worker that previously registered with the cluster but stopped advertising itself for some reason. Usually this is due to network connectivity issues or due to the worker stopping unexpectedly. 

2. The workers have run out of disk space.

Environment

Product Version: 6.3

Resolution

As a workaround, you can check for stalled workers by running fly workers and pruning any stalled workers with fly prune-worker -a which would prune all stalled workers. You can use the -w option instead of -a to prune a specific worker. 

If pruning the workers does not resolve the issue, you can look at the disk space on the workers and resize them accordingly.

In addition, you can also try recreating the worker VMs.

If after trying the steps above you are still faced with the same issue, please open a support ticket with VMware Tanzu and upload the logs from the web and worker VMs.