'target is busy' error blocks umount command, Apply Changes fails

Products

• VMware Tanzu Application Service
• VMware Tanzu Application Service for VMs
• VMware Tanzu Application Service

Issue/Introduction

Customer was following another KB to alleviate a full filesystem on a VM. After this, they wanted to enlarge the persistent disk. They doubled the size of the disk in the TAS Resource Config, but when applying changes they encountered this error:

Task 779340 | 20:09:02 | L stopping jobs: mysql/########-####-####-####-############ (1) (canary)
Task 779340 | 20:09:04 | L executing post-stop: mysql/########-####-####-####-############ (1) (canary) (00:10:44)
                      L Error: Action Failed get_task: Task########-####-####-####-############ result: Migrating persistent disk: Remounting persistent disk as readonly: Unmounting /var/vcap/store: Running command: 'umount /var/vcap/store', stdout: '', stderr: 'umount: /var/vcap/store: target is busy.

Environment

Tanzu Application Service for VMs
Ubuntu 22.04 long-term support (LTS) stemcell, Jammy Jellyfish

Cause

An interactive session where the user has changed directories (cd) into a filesystem will cause that filesystem to be busy in the view of the operating system, thus causing the umount command to fail.

Resolution

By leaving the filesystem, either by changing to another directory outside of that filesystem or by terminating the login session, the user frees up the filesystem. 

The lsof command can also help to identify any processes which may be interacting with a particular filesystem and making it busy.