Fail to upgrade stemcell with error "unable to save stemcell assignments" in Ops Manager UI

Products

• Operations Manager

Issue/Introduction

After having upgraded to a new version of Ops Manager, the stemcells for some service tiles might be showed as "outdated". As an attempt to keep the stemcells latest, we click "Save" button in Ops Manager UI after picking up the newer stemcell. An error might pop out like the following:

Environment

Product Version: 2.10

Resolution

When export/import an Ops Manager installation, we don’t export those source tiles and stemcells. So after importing an installation, we need to re-upload the active tiles and stemcells into Ops Manager again. 

The following procedures are an example to fix this issue.
1. Log in the ops manager vm

2. Review the /var/log/opsmanager/audit_log.txt and filter it with keyword, "stemcell_associations" and "422".

Then you might see the messages like the following ones:

grep "stemcell_associations" audit_log.txt|grep 422

{"timestamp":"xxx","component":"Tanzu Ops Manager xxx","component_scheme":"https","component_host":"xxx","component_port":443,"result":422,"target":"/api/v0/stemcell_associations","event_type":"PATCH","origination":"xxx","forwarded_for":"xxx","user_identification":"pselvam","uaa_authentication_mechanism":"ldap","request_body":{"products":[{"guid":"p-tilename-xxx","staged_stemcells":[{"os":"ubuntu-xenial","version":"456.158"}]}],"stemcell_association":{"products":[{"guid":"p-tilename-xxx","staged_stemcells":[{"os":"ubuntu-xenial","version":"456.158"}]}]}},"reason":"{\"errors\":[\"p-tilename-xxx: stemcell (os: ubuntu-xenial, version: 456.158) does not exist\"]}"}

3. The error above often occurs due to the stemcell not being present in the /var/tempest/stemcells directory on the Ops Manager VM.

4. Download the missing stemcells from the pivnet and upload to the foundation
For example:

bosh -e xxx upload-stemcell bosh-stemcell-456.158-vsphere-esxi-ubuntu-xenial-go_agent.tgz --fix