`bosh create-env` command results in error "cannot create directory" due to "File exists"

Products

• Operations Manager

Issue/Introduction

The issue occurs when manually recreating or upgrading BOSH director with BOSH CLI `create-env`. The full error logs are as below:  

$ bosh --non-interactive --tty create-env ...
Deployment state: 'state.json'

Started validating
  Validating release 'bosh'... Finished (00:00:06)
  Validating release 'bpm'... Finished (00:00:02)
  Validating release 'bosh-vsphere-cpi'... Finished (00:00:05)
  Validating release 'os-conf'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Validating stemcell... Finished (00:00:05)
Finished validating (00:00:16)

Started installing CPI
  Compiling package 'golang-1-darwin/ef41b54400818371a122c2d72abcc0c2e9b8c2eb59a4db25fe8564a64b55ffb6'... Failed (00:00:28)
Failed installing CPI (00:00:28)

Error: Installing CPI:
  Compiling job package dependencies for installation:
    Compiling job package dependencies:
      Compiling package:
        Running command: 'bash -x packaging', stdout: '', stderr: '+ set -e -x -u
+ tar xzf go1.22.5.darwin-amd64.tar.gz
+ cp -R go/CONTRIBUTING.md go/LICENSE go/PATENTS go/README.md go/SECURITY.md go/VERSION go/api go/bin go/codereview.cfg go/doc go/go.env go/lib go/misc go/pkg go/src go/test /home/ubuntu/.bosh/installations/7fb80686-5ebc-4637-41dd-4af7817d81d6/packages/golang-1-darwin
+ mkdir /home/ubuntu/.bosh/installations/7fb80686-5ebc-4637-41dd-4af7817d81d6/packages/golang-1-darwin/bosh
mkdir: cannot create directory '/home/ubuntu/.bosh/installations/7fb80686-5ebc-4637-41dd-4af7817d81d6/packages/golang-1-darwin/bosh': File exists
':
exit status 1

Environment

BOSH CLI all versions

Cause

During "installation CPI" step of `bosh create-env` execution, the CLI creates directory ~/.bosh/installations/<INSTALLATION_ID> and put CPI packages as well as compiled packages inside. In the meanwhile, compiled_packages.json file is created to track the compiled packages. In the case of BOSH director redeployment with `bosh create-env`, the CLI will skip compilation as the compiled packages already exist.

In the case of BOSH director upgrade, BOSH CLI can find new packages which are not compiled yet, thus it will run compilation step as expected. In the case of above go1.22.5.darwin-amd64.tar.gz compilation, BOSH CLI will create a directory ~/.bosh/installations/<INSTALLATION_ID>/packages/golang-1-darwin/bosh as specified in package script. However the directory has been created in previous deployment and it is not deleted automatically for cache purpose.

Resolution

To resolve the issue, please remove ~/.bosh/installations/<INSTALLATION_ID> directory before a deployment of new BOSH director version. It is not needed to remove the directory for redeployment of same BOSH director version as the directory could help to same deployment time by skip compiled packages.