Updating BOSH Director fails with "Could not complete network copy for file"

Products

• VMware Tanzu Application Service
• Operations Manager

Issue/Introduction

When attempting to run an Apply Changes against the BOSH Director, it fails with:

Started deploying
  Creating VM for instance 'bosh/0' from stemcell 'sc-xxx'... Failed (00:02:08)
Failed deploying (00:02:08)

Cleaning up rendered CPI jobs... Finished (00:00:00)


Deploying:
  Creating instance 'bosh/0':
    Creating VM:
      Creating vm with stemcell cid 'sc-xxx':
        CPI 'create_vm' method responded with error: CmdError{"type":"Unknown","message":"Could not complete network copy for file /vmfs/volumes/<id>/sc-xxx/sc-xxx-000001.vmdk","ok_to_retry":false}

Environment

TPCF deployed on vSphere

Cause

This is an error from vSphere, claiming it cannot clone the stemcell VM to create the new BOSH Director. The same error can be validated on vSphere -> sc-xxx (from error message) -> Monitor -> Tasks

Resolution

To resolve this, we can force Ops Manager to deploy a new stemcell VM. To do this:

1. SSH to Ops Manager
2. cd /var/tempest/workspaces/default/deployments
3. sudo cp bosh-state.json /tmp/bosh-state-backup.json
4. sudo vim bosh-state.json
5. Under "stemcells", remove the entry for the stemcell in the error message. For example, change from this:

    "stemcells": [
        {
            "id": "<id>",
            "name": "bosh-vsphere-esxi-ubuntu-jammy-go_agent",
            "version": "1.844",
            "api_version": 3,
            "cid": "sc-xxx"
        }
    ],

         To this:

    "stemcells": [
    ],

      6. Once this is done, Apply Changes again. Ops Manager will deploy a new stemcell VM to vSphere, and allow the changes to progress.

 

If you face this issue at any stage of the apply changes other than BOSH Director, it can be fixed by reuploading the stemcell with --fix. For example:

bosh upload-stemcell /var/tempest/stemcells/bosh-vsphere-esxi-ubuntu-jammy-go_agent-xxx.tgz --fix