How to add or modify my availability zones without the need to reinstall Tanzu Application Services

Products

Operations Manager

Issue/Introduction

When you initially install Tanzu Operations Manager and Tanzu Application Services, you’ll be asked to define the availability zones (AZs) for the installation.

These dictate where the platform will place Virtual machines (VMs) on your IaaS. After you perform the initial installation, Ops Manager will lock these fields, and it won’t let you make changes to them. For additional information on locked fields see the docs .

This article describes the process to go through if you need to change your availability zone configuration after the initial install.

Environment

Product Version: 2.10

Resolution

Before making any changes, it’s important to realize that there are some limitations in what you can do.

Migrating singleton jobs ( VMs that have only 1 instance ) will result in some downtime or data loss if a persistent disk is used. Make sure you have a strategy for backing up and migrating any persistent data that will be lost as a result of the bosh AZ migration. One example of a singleton job is the nfs_server vm. Migrating the nfs_server VM will result in complete loss of the cf droplet blobstore unless you have a restore strategy post migration.
Any non-singleton ( vms with multiple instances ) jobs that have persistent disks will also result in data loss. BOSH does not migrate persistent disk data between AZs. Any VM that has persistent data requirements will experience data loss during the migration. The exception is the data can be restored or regenerated, like MySQL where it can synchronize with another node in the cluster and restore the persistent data (this assumes cross-AZ network connectivity) or the Diego Database where it can rebuild the database from scratch.
Moving VMs out of an AZ may lead to IP allocation issues. BOSH does not release allocated IPs until after a successful deployment has been achieved. For example if your subnet range allows 64 ips and there are 40 vms that need to be migrated then bosh will require 85 IPs to complete the migration ( 40 old ips, 40 new ips, and 5 additional IPs for compilation vms ). You can work around this by scaling the job instance count down, adding the new AZ, and then scaling the job instance count back up. Please keep in mind that scaling down the instance count can have its own problems, such as causing capacity problems or causing a job that is usually clustered like MySQL to no longer be clustered. Make sure you consider the impact of this before scaling down the job instance count.
Renaming an AZ is basically handled like an AZ migration, which means all of the stipulations listed above would apply for a rename. Using generic names like `AZ1`, `AZ2`, `AZ3`, etc. is recommended. If you name after underlying hardware or infrastructure, the names may not make sense as the underlying infrastructure evolves.
Validate Memory Capacity Because of the drastic scaling of the cells when an AZ is added, the recommended approach for any change is to have Diego cell memory usage under 33% for a 3AZ scenario. If this capacity is not available, a scenario where cell memory capacity gets drained to 0 can cause some app instances to remain shut down until the new Diego cell instances get updated.

In order to make changes to your Availability Zone configuration in Operations (Ops) Manager, you will need to follow procedures for modified lock on deploy fields. First you must enable Advanced mode in Operations Manager following the instructions in this article or see below Modify AZ configurations using Operations Manager API. When Operations Manager is in Advanced mode ( otherwise known as unlocked ) there are limitations as to which fields can be edited in the UI and API. Below explains the limitations of each.

Limitations Editing AZ configuration using Operations Manager UI

We recommend using the UI for all AZ migration changes.

Operators will be able to Delete AZs that are not used by any tile. If there is a Tile still using the AZ then you will not be able to delete the AZ.
Operators can Change Tile AZ configurations. For example if you have AZ1, AZ2, A3, and add a new AZ4. You could uncheck AZ1 from the Tile config and check AZ4. This would effectively migrate vms from AZ1 to AZ4
VSphere Clusters within an AZ are immutable and can not be modified in any way.

Limitations Editing BOSH AZ configurations using Operations Manager API ( om cli )

Ideally used only when making modifications to vSphere cluster configurations within AZs. This method is an advanced procedure which can result in unknown consequences if not used as intended.

Operators can Rename vSphere Clusters within an AZ
Operators can remove clusters from AZ. This is commonly used when upgrading vSphere cluster hardware from old to new. Simply add a new cluster via the UI and remove the old cluster using the API.

Modify AZ configurations using Operations Manager API

Set the following environment variables

export OM_TARGET=OPSMAN_HOSTNAME
export OM_USERNAME=admin
export OM_SKIP_SSL_VALIDATION=true
export OM_PASSWORD=PASSWORD

Unlock advanced mode

om curl -p /api/v0/staged/infrastructure/locked -x PUT --data '{"locked" : "false"}' -H "Content-Type: application/json"

Get existing availability zones (Ops Manager API does not concatenate values when making a change via API)
- ```
om curl -p /api/v0/staged/director/availability_zones > azs.json
```
Modify azs.json

Update AZ configs in Operations Manager

om curl -p "/api/v0/staged/director/availability_zones" -x PUT -H "Content-Type: application/json" -d @azs.json

Lock Opsman to disable Advanced Mode

om curl -p /api/v0/staged/infrastructure/locked -x PUT --data '{"locked" : "true"}' -H "Content-Type: application/json"

Verify the locked status

om curl -p /api/v0/staged/infrastructure/locked

Other Tile AZ migration Considerations

This article describes the procedure to migrate Redis On-Demand Services instances from one AZ to another.