Unable to Copy Stemcell Root Image - AWS Bosh Director

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

When attempting to upload a heavy stemcell to an AWS foundation via Tanzu Operations Manager, the create_stemcell CPI operation fails with the following error:

CPI 'create_stemcell' method responded with error:
CmdError{
  "type":"Bosh::Clouds::CloudError",
  "message":"Unable to copy stemcell root image:
    command 'sudo -n /usr/local/bin/stemcell-copy
      <tmp-path>/image
      /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_<vol-id> 2>&1'
    failed with exit code 1
    Script output:
      sudo: /etc/sudo.conf is owned by uid 65534, should be 0
      sudo: error in /etc/sudo.conf, line 0 while loading plugin "sudoers_policy"
      sudo: /usr/libexec/sudo/sudoers.so must be owned by uid 0
      sudo: fatal error, unable to load plugins",
  "ok_to_retry":false
}

Environment

IaaS: AWS
Product: Tanzu Operations Manager (all versions)

Cause

AWS stemcells are distributed in two formats:

• Light stemcells contain only a manifest with pre-built AMI IDs per region. When uploaded, the BOSH CPI simply returns the matching AMI ID without performing any disk operations.
• Heavy stemcells contain a full raw disk image. When uploaded, the AWS CPI must write this image to an EBS volume by invoking the stemcell-copy script with root privileges via sudo.

Ops Manager supports heavy stemcells for all VMs except the BOSH Director VM. This is an incompatibility and cannot be resolved by modifying file permissions on the Ops Manager VM.

Resolution

The workaround is to bypass the stemcell-copy operation entirely by pre-importing the raw disk image into your AWS account as an AMI, then constructing a custom light stemcell manifest that references that AMI directly. The BOSH Director will use the AMI ID from the manifest and never invoke stemcell-copy.

1. Download Heavy Stemcell
2. Extract the Raw Disk Image from Heavy Stemcell
3. Create an S3 Bucket for Import
4. Upload Raw Disk Image to S3 Bucket
5. Import the Image as an EBS Snapshot
6. Create a temporary EC2 Instance from the Snapshot
7. Stop the Instance
8. Create the AMI from the Instance
9. Create the Custom Light Stemcell .tgz that points to your AMI (light stemcell only contains metadata, not raw image)
10. Upload Light Stemcell to OpsMan
11. Apply Changes