Scaling up HAProxy VM in VMWare TAS does not improve system performance

Products

• VMware Tanzu Application Service for VMs

Resolution

Checklist:

Both VMware Tanzu Application Service(TAS) for VMs and Isolation Segment include HAProxy which provides load balancing for the Gorouters.

The HAProxy job on the HAProxy VM is configured to use only a single CPU, thus under heavy loads, even if there are multiple CPUs available with the VM, utilization of one CPU could spike to 100%, while other CPUs are almost idle. 

03:45:01   CPU     %user     %nice   %system   %iowait    %steal     %idle
03:45:01   all     49.33      0.00      1.91      0.01      0.00     48.75
03:45:01     0      4.43      0.00      1.53      0.03      0.00     94.01
03:45:01     1     93.77      0.00      2.29      0.00      0.00      3.94

Both the Ops Manager status page and `bosh -is --vitals` only returns total CPU utilization and not per CPU usage. For example in the case of two CPUs with HAProxy VM and heavy load, you'd see 50% CPU utilization indicated. This means the HAProxy is completely utilizing the single CPU it's configured to use.

Given this limit, vertical scaling up of HAProxy VMs doesn't provider additional capacity to HAProxy process and should be avoided. HAProxy VMs should be horizontally scaled by adding more instances.

Additional Information

While HAproxy as configured by TAS and the Isolation Segment tile cannot be scaled vertically (i.e. by adding RAM/CPU), it can be scaled horizontally (i.e. by adding more instances) and thus can achieve high availability and can be used in production scenarios, when deployed with multiple instances.