Upload stemcell fails with error "Cannot find EBS volume on current instance"

Products

• Operations Manager

Issue/Introduction

Symptoms:

upload-stemcell fails with the error "Cannot find EBS volume on current instance".

The operator cannot upload heavy stemcells to AWS. The upload-stemcell task fails with the following message:

bosh --environment <env> upload-stemcell <stemcell file>

This problem can occur if the operator is defining custom VM types for the BOSH director.

BOSH task <task id> --cpi shows stack trace:

[2019-06-06T18:03:55.014804 #26599] ERROR -- [req_id cpi-878737]: Cannot find EBS volume on current instance (Bosh::Clouds::CloudError) /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/lib/cloud/aws/helpers.rb:11:in `cloud_error' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/lib/cloud/aws/cloud_v1.rb:441:in `find_device_path_by_name' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/lib/cloud/aws/cloud_v1.rb:481:in `create_ami_for_stemcell' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/lib/cloud/aws/cloud_v1.rb:387:in `block in create_stemcell' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/vendor/bundle/ruby/2.4.0/gems/bosh_common-1.3262.24.0/lib/common/thread_formatter.rb:49:in `with_thread_name' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/lib/cloud/aws/cloud_v1.rb:356:in `create_stemcell' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/vendor/bundle/ruby/2.4.0/gems/bosh_cpi-2.5.0/lib/bosh/cpi/cli.rb:89:in `public_send' /var/vcap/data/packages/bosh_aws_cpi/6f6993455b30e53013ff33dce753322dd137a64b/vendor/bundle/ruby/2.4.0/gems/bosh_cpi-2.5.0/lib/bosh/cpi/cli.rb:89:in `run' /var/vcap/packages/bosh_aws_cpi/bin/aws_cpi:28:in `<top (required)>'

Environment

Cause

This problem is caused by a bug in the AWS CPI.

See this tracker story for more details.

The issue is due to a defect in finding devices attached to M5 VM type. As part of uploading a stemcell, the BOSH director must attach a volume temporarily to create an Amazon Machine Image (AMI) for the new stemcell.

Resolution

The solution is to switch your BOSH director so that it does not use the M5 VM type line on AWS.

This requires updating your bosh.yml to change the instance type to M4 or an older version:

bosh.yml:
cloud_properties:
   instance_type: m5.xlarge -> switch to m4.xlarge

Next you will need to run bosh create-env to recreate your BOSH director. The upload-stemcell commands will succeed once the director is recreated.

Alternatively, if you are using Operations Manager with custom VM types, the solution is to select a different VM type (not M5) in OpsManager > Director > Resource Config, then click Apply Changes.