Platform Automation task Download-Product fails when putting the file on S3

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When the operator uses the task Download-Product to download a product from the Tanzu Network (Pivnet) and store the file on a S3 compatible blobstore, the PUT/GET task fails with one of the following errors:

Error 1:

error running command: regex does not match provided version: s3resource.Version{Path:"<product and version>", VersionID:""}

Error 2:

error running command: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

Error 3:

panic: version number was not valid: Expected version 'XXXXX]<product-version>' to match version format

goroutine 1 [running]:
github.com/concourse/s3-resource/versions.Extract(0xc0000d6600, 0x40, 0xc000014d80, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/concourse/s3-resource/versions/versions.go:57 +0x30c
github.com/concourse/s3-resource/in.(*Command).Run(0xc000159dc8, 0x7ffc2ecc2eac, 0xe, 0xc000014d60, 0x14, 0xc0000163c0, 0x28, 0x0, 0x0, 0xc0000132f0, ...)
	/go/src/github.com/concourse/s3-resource/in/command.go:69 +0x133
main.main()
	/go/src/github.com/concourse/s3-resource/cmd/in/main.go:61 +0x2ea

Resolution

All three of these errors are due to configuration issues on either the S3 resource or in the download-product config file. 

When uploading a product file to a S3 compatible blobstore, there is an explicit PUT request being passed and per design, every time a PUT is passed a GET will also be triggered on the new resource created. So there is a need to make sure that the S3 resource is configured to perform a GET. The following resource contains the source parameters needed for a PUT and a GET. 

- name: <resource name>
  type: s3
  source:
    endpoint: ((blobstore_endpoint))#needed when not using Amazon S3 (e.g: Minio)
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp:

Resolution of Error 1:

Error 1 is the most common out of the three specified in this list. This error appears when the S3 blobstore bucket is not defined in the download-product config file. Tanzu Network filenames will not always contain the necessary metadata to accurately download files from a blobstore (i.e. s3, gcs, azure). Therefore, it is required that the operator adds the blobstore-bucket flag to the config file so that the product slug and version are prepended when using the download-product task.

---
pivnet-api-token: token
pivnet-file-glob: "*.pivotal"       # must be quoted if starting with a *
pivnet-product-slug: product-slug

# Either product-version OR product-version-regex is required
# product-version-regex: ^1\.2\..*$ # must not be quoted
product-version: 1.2.3
blobstore-bucket: bucket  # if set, product files will have their slug and
                            # version prepended. Set if the product will
                            # ever be stored in a blobstore

Resolution of Error 2:

Error 2 shows when the S3 resource defined in the pipeline manifest does not include an entry for the regular expression. Please ensure that the regexp is part of the S3 resource parameters since it's needed to perform the GET. 

Resolution of Error 3:

Error 3 means that the user included the regex flag to the S3 resource but it's using the wrong format. 
The following doc has examples of what the format looks like: https://docs.pivotal.io/platform-automation/v4.4/pipelines/resources.html#product-resources