vSphere memory-based cluster balancing

Products

• Operations Manager

Issue/Introduction

Introduction

This document describes the steps need to configure Tanzu Operations Manager to make optimal use of multi-asymmetrical vSphere clusters per AZ. In this configuration, cluster balancing will be controlled by using full memory reservations on VMs and setting memory reservations on the Resource Pools in the clusters.

BOSH will optimistically try one of the first cluster candidates when creating a VM and when it fails to start the VM due to not enough resources being available it will gracefully try the next cluster.

Starting from VMware Tanzu Ops Manager version 2.10.49 (vSphere CPI version 85 and above), “vm_extension” need to be set to enable this memory-pinning for the VMs created by BOSH.

Environment

Product Version: 2.10

Resolution

Configuration Steps

Configure memory pinning via BOSH vm_extension

In BOSH vSphere cpi 85 (included in VMware Tanzu Ops Manager 2.10.49), a new property has been added to allow locking the memory reservation to the max memory configured for the VM.

To use this feature an additional property must be passed into cloud_properties on VM creation. The mechanism BOSH uses for this is called vm_extensions, these can be configured in Tanzu Operation Manager via the API (not currently possible through the graphical user interface).

Prerequisites

To manage custom VM extensions, you need the following:

• Access to the Ops Manager API: For more information about the Ops Manager API, see Using the Ops Manager API.
• The om cli needs to be installed, see installation instructions.
• The following environment variables need to be populated with the appropriate credentials / endpoint to be able to successfully interact with the Ops Manager:
  • OM_TARGET=https://opsman.example.com
  • OM_USERNAME=username
  • OM_PASSWORD=password

Add a vm_extension

Execute the following snippet to create a vm_extension called pin_memory:

om curl -p /api/v0/staged/vm_extensions/pin_memory -x PUT \

-d '{"name": "pin_memory", "cloud_properties": {

"memory_reservation_locked_to_max": true }}'

Apply vm_extension to all instance types

The following snippet will grab the staged config for the cf product. Convert it from yaml to json using ruby. Append the pin_memory to each entry under resource-config via additional_vm_extensions. This config is then submitted to the Ops Manager via configure-product.

om configure-product -c <( \

om staged-config -p cf | \

ruby -ryaml -rjson -e 'puts JSON.generate(YAML.load(ARGF))' | \

jq '.["resource-config"] |= 

with_entries(.value.additional_vm_extensions += ["pin_memory"])')

Alternatively, you can use yq to convert yaml to json as per the following snippet:

om configure-product -c <( \

om staged-config -p cf | \

yq r -j - | jq '.["resource-config"] |= 

with_entries(.value.additional_vm_extensions += ["pin_memory"])')

These instructions are for the TAS (cf) tile and should be repeated for other tiles as well.

On demand tiles

TKGi: To apply vm_extensions on TKGi clusters please follow these docs.

Tanzu RabbitMQ for TAS [VMs]: add the pin_memory extension under List of VM Extensions for each service plan.

Tanzu GemFire for TAS [VMs]: at the time of writing this document it is not yet possible to configure VM extensions for this tile.

Apply Changes

With the cpi in place and the vm_extensions configured it is now time to apply changes. The cpi changes will not show up in the UI, but the vm_extensions should be visible. Given that all cloud_properties of the VMs have changed, BOSH will go through a recreation of every instance (just like when updating a stemcell).