Best approach during object storage maintenance activity

Products

• Operations Manager

Issue/Introduction

We have maintenance activity scheduled for our s3 compatible blob store object storage. During this maintenance activity, the object storage will completely be in a shutdown state and will not be accessible and this will affect our Tanzu Application Service (TAS) foundation. Therefore, we would like to understand what the best practice / best approach would be in this scenario - will we need to shut down specific TAS component VMs?

Environment

Product Version: 2.7

Resolution

File Storage should be HA and we recommend against shutting it down even for a short period of time, but if you need to do it, here are some tips to follow:

• If the File Storage holds any of the following databases: Director DB, TAS UAA DB, TAS Credhub DB, or TAS DB, then the whole foundation should be shut down during the maintenance period. If that is not the case, see following points.
• If the File Storage holds the BOSH director blobstore, bosh won't be able to recover any failing VM so bosh resurrector should be disabled to avoid bosh continuously failing in that task.
• If the File Storage holds TAS blobstore, any app that crashes will fail to be recovered by Diego and there isn't anything you can do to avoid that. All apps will be recovered once the File Storage is up and running again. In addition, any restart, restage, or any new push will fail so our recommendation would be to ask your developers not to perform any of those operations.
• If the File Storage holds your backups, stop any scheduled backups that may be triggered during the maintenance period.