Ops Manager BOSH Director creation fails on AWS with "Unable to copy stemcell root image" / sudo.conf ownership error when using a heavy stemcell

Products

• VMware Tanzu Platform Core
• VMware Tanzu Platform
• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

When upgrading or installing Tanzu Operations Manager (Ops Manager) on AWS, the BOSH Director creation step fails while creating the stemcell, with an error similar to:

Creating stemcell (bosh-aws-xen-hvm-ubuntu-jammy-go_agent 1.1193):
  CPI 'create_stemcell' method responded with error: CmdError{"type":"Bosh::Clouds::CloudError","message":"Unable to copy stemcell root image: command 'sudo -n /usr/local/bin/stemcell-copy ... /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol... 2>&1' failed with exit code 1
Script output:
sudo: /etc/sudo.conf is owned by uid 65534, should be 0
sudo: error in /etc/sudo.conf, line 0 while loading plugin "sudoers_policy"
sudo: /usr/libexec/sudo/sudoers.so must be owned by uid 0
sudo: fatal error, unable to load plugins
","ok_to_retry":false}

This occurs during the "Creating stemcell" phase of BOSH Director deployment, before any other tiles or VMs are created.

Cause

Ops Manager supports two stemcell formats on AWS:

• Light stemcells — reference an existing published AWS AMI; no image copy is required.
• Heavy stemcells — ship the full root image, which the AWS CPI must copy onto a raw EBS volume via a helper script (stemcell-copy) before the Director VM/AMI can be created. Because the tempest-web user that runs this step does not otherwise have the privileges needed for this operation, Ops Manager grants it narrowly-scoped sudo access to just this script.

Starting in Ops Manager 3.1, the bosh CLI on the Ops Manager VM was wrapped to prevent it from deleting BOSH releases shipped as part of Foundation Core. This wrapper change inadvertently broke the sudoers configuration backing the stemcell-copy script, causing sudo itself to fail to load its plugins (/etc/sudo.conf and sudoers.so ownership errors) whenever a heavy stemcell is used for BOSH Director creation.

This is a packaging regression in Ops Manager, not a corrupted VM, disk, or misconfiguration. It affects Ops Manager 3.1, 3.2, and 3.3 whenever a heavy stemcell is used for the Director. 

Resolution

Use a light stemcell (the standard AWS AMI-referencing stemcell published by BOSH) for the BOSH Director instead of a heavy stemcell. This avoids the broken stemcell-copy sudo path entirely and allows Director creation to complete successfully.

Steps:

1. In the Ops Manager VM under /var/tempest/workspaces/default/deployments edit the file bosh-state.json  and remove the stemcell reference as below example snippet:
   
   

   Before:
       "stemcells": [
           {
               "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
               "name": "bosh-vsphere-esxi-ubuntu-jammy-go_agent",
               "version": "1.1183",
               "api_version": 3,
               "cid": "sc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
           }
       ],
   
   
   After 
       "stemcells": [ ],

2. Make sure the stemcell AMI defined under the stemcell.MF file under the /var/tempest/stemcells/stemcell.tgz file is present in your region, if not copy the AMI from one region to another in AWS console, make a note of the new AMI ID and then update in the stemcell.MF file.
3. Re-run Apply Changes / retry the upgrade.
4. Confirm the "Creating stemcell" step completes without the sudo/stemcell-copy error.