SCS tiles Apply Change failed with error message "'umount: /var/vcap/store: target is busy."

Products

• VMware Tanzu Application Service

Issue/Introduction

The customer upgrades SCS tile with stemcell update. Apply Change might be failed at post-stop step due to the Git operations (initiated by mirror refresh) were keeping the persistent disk busy so that unmount failed. 

Error message as below -

  stemcells:
  - name: bosh-vsphere-esxi-ubuntu-jammy-go_agent
-   version: '1.866'
+   version: '1.894'
  
  releases:
  - name: bosh-dns-aliases
-   file: bosh-dns-aliases-0.0.4-ubuntu-jammy-1.719.tgz
+   file: bosh-dns-aliases-0.0.4-ubuntu-jammy-1.866.tgz
Task #####

Task ##### | 05:46:52 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task ##### | 05:46:52 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task ##### | 05:46:52 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task ##### | 05:46:52 | Deprecation: Top level 'properties' in addons are deprecated. Please define 'properties' at the job level.
Task ##### | 05:46:53 | Preparing deployment: Preparing deployment (00:00:03)
Task ##### | 05:46:56 | Preparing deployment: Rendering templates (00:00:01)
Task ##### | 05:46:57 | Preparing package compilation: Finding packages to compile (00:00:00)
Task ##### | 05:47:09 | Updating instance spring-cloud-services: spring-cloud-services/##### (0) (canary)
Task ##### | 05:47:10 | L executing pre-stop: spring-cloud-services/##### (0) (canary)
Task ##### | 05:47:15 | L executing drain: spring-cloud-services/##### (0) (canary)
Task ##### | 05:47:16 | L stopping jobs: spring-cloud-services/##### (0) (canary)
Task ##### | 05:47:51 | L executing post-stop: spring-cloud-services/##### (0) (canary) (00:10:50)
                        L Error: Action Failed get_task: Task ##### result: Unmounting persistent disk: Running command: 'umount /dev/sdc1', stdout: '', stderr: 'umount: /var/vcap/store: target is busy.
Updating deployment:
  Expected task '#####' to succeed but state is 'error'
': exit status 32
Task 6319800 | 05:57:59 | Error: Action Failed get_task: Task ##### result: Unmounting persistent disk: Running command: 'umount /dev/sdc1', stdout: '', stderr: 'umount: /var/vcap/store: target is busy.
': exit status 32

 

Environment

Spring Cloud Service Tile v3.3.x

Cause

The root cause is until the last moment that the SCS VM is up and running, ssh connections are also going through. So service instances can keep calling git fetch all this time during upgrade. An slow git fetch right at the end of post-stop or during post-stop step, can keep the persistent disk busy thus causing the failure.

Resolution

Follow below steps to resolve the issue -

1. SSH to the SCS broker VM and run "sudo shutdown now"
2. Run the below bosh command "bosh -d SCS_DEPLOYMENT_NAME deploy /var/tempest/workspaces/default/deployments/SCS_DEPLOYMENT_NAME.yml --fix"
3. Apply Change from OpsManager for SCS tile alone