Concourse Pipeline Failure: "stream image metadata file: file not found" when upgrading cf-mgmt via S3 image

Products

VMware Tanzu Platform Core Concourse for VMware Tanzu

Issue/Introduction

When upgrading the cf-mgmt utility in a Concourse pipeline from an older open-source version (e.g., 1.0.102) to a newer release downloaded from the Broadcom support portal (e.g., starting from1.0.107 and above), the pipeline task fails immediately upon initialization.

The task uses an S3 bucket as the container execution environment via the image_resource block:

image_resource:
  type: s3
  source:
    ...
  params:
    unpack: true

Upon executing the updated pipeline, the Concourse worker returns the following error:

stream image metadata file: file not found

Note: If a user attempts to temporarily revert the pipeline back to the older version by changing the S3 bucket contents without hardcoding the version regex, they may experience a secondary Concourse caching error: version is missing from previous step.

Environment

Concourse
cf-mgmt
Tanzu Platform

Cause

This issue occurs because of a change in how the release artifacts are packaged, combined with how Concourse processes S3 image resources.

When type: s3 is used with unpack: true under an image_resource, Concourse does not simply extract the tarball as a file asset; it attempts to boot the .tgz file as a complete container operating system. For Concourse to boot an archive successfully, the .tgz must be formatted as a container root filesystem containing two specific elements at its root:

A metadata.json file.
A rootfs/ directory containing a base Linux OS structure (e.g., /bin, /etc, /usr).

The Packaging Discrepancy:

Older versions (Custom/Pre-packaged in upstream): The .tgz previously uploaded to the S3 bucket for 1.0.102 was custom-packaged to include this full root filesystem and metadata.
Newer versions (Broadcom Portal): From 1.0.107 release from Broadcom provides only the raw compiled binaries (e.g., cf-mgmt-linux and cf-mgmt-config-linux). It lacks the rootfs/ and metadata.json.

Concourse fails because it is attempting to boot a tarball containing two bare executable files as if it were a full Linux operating system.

Resolution

To resolve this issue while maintaining the S3 image_resource configuration, you must manually package the new Broadcom binaries into a Concourse-compatible OS tarball before uploading it to S3.

The following steps outline how to generate a clean Ubuntu-based root filesystem containing the updated cf-mgmt binaries using Docker.

Step 1: Export a clean Linux filesystem

On a local machine with Docker installed, pull a lightweight base image (like Ubuntu) and extract its filesystem structure:

# Create a workspace
mkdir -p /tmp/cf-mgmt-image/rootfs
cd /tmp/cf-mgmt-image

# Pull and extract the container filesystem
docker pull ubuntu:latest
docker create --name temp-ubuntu ubuntu:latest
docker export temp-ubuntu | tar -xC rootfs/
docker rm temp-ubuntu

Step 2: Inject and rename the new binaries

Extract the Broadcom 1.0.109 archive locally. Copy the binaries into the standard executable path of the newly created Ubuntu filesystem.

# Replace /path/to/extracted/broadcom/archive/ with your actual path
cp /path/to/extracted/broadcom/archive/cf-mgmt-linux rootfs/usr/local/bin/cf-mgmt
cp /path/to/extracted/broadcom/archive/cf-mgmt-config-linux rootfs/usr/local/bin/cf-mgmt-config

# Ensure the binaries are executable
chmod +x rootfs/usr/local/bin/cf-mgmt*

Step 3: Create the Concourse metadata file

Generate the metadata.json file required by Concourse to boot the archive:

cat <<EOF > metadata.json
{
  "user": "root",
  "env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
}
EOF

Step 4: Package and Upload to S3

Compress the metadata.json file and the rootfs/ directory into a final .tgz archive:

tar -czf cf-mgmt-1.0.109.tgz metadata.json rootfs/

Upload this newly created cf-mgmt-1.0.109.tgz file to the S3 bucket path expected by your pipeline (e.g., pivotalservices/cf-mgmt-1.0.109.tgz).

The Concourse worker will now successfully parse the metadata, boot the Ubuntu rootfs, and execute the updated cf-mgmt binaries natively.