How to increase GoRouter buffer ring size

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

The default buffer ring size for TAS VMs is 256. Sometimes due to high load, this can overflow on gorouters VMs which causes delays in response. We can increase buffer ring size in gorouters VMs only

Environment

Product Version: 2.11

Resolution

Procedure to increase buffer ring size in gorouters VMs only.
ethtool that is used to modify the buffer ring size is now pre-installed in the Xenial stemcell v621.364+ and Jammy stemcell v1.64+


1 - upload the os-conf release to bosh:

bosh upload-release https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=22.1.2

2 - Create os-conf.yml:

releases:
- name: os-conf
  version: 22.1.2
  url: https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=22.1.2
  sha1: 386293038ae3d00813eaa475b4acf63f8da226ef
addons:
  - name: os-configuration
    include:
      instance_groups:
      - router
      deployments:
      - cf-1b44df45630375c5d1df
    jobs:
    - name: pre-start-script
      release: os-conf
      properties:
        script: |-
          #!/bin/bash
          set -ex
          RX_SIZE=1024
          RX_JUMBO=256
          /sbin/ethtool -G eth0 rx $RX_SIZE rx-jumbo $RX_JUMBO
          echo "ACTION==\"add|change\", SUBSYSTEM==\"net\", KERNEL==\"eth*|en*\", RUN+=\"/sbin/ethtool -G \$name rx ${RX_SIZE} rx-jumbo ${RX_JUMBO}\"" > /etc/udev/rules.d/61-net.ring.rules

3 - Upload the runtime-config to bosh:

bosh update-config --type=runtime --name=os-conf-ringbuffer os-conf.yml

 4 - Deploy (Apply Changes)
Once this deploy finished we could see that the settings and script were on router instances:

router/61d828bd-82eb-4296-a231-73a22aad3dfc:/var/vcap/jobs/pre-start-script/bin# ls
pre-start
router/61d828bd-82eb-4296-a231-73a22aad3dfc:/var/vcap/jobs/pre-start-script/bin# cat pre-start
#!/bin/bash

set -ex

RX_SIZE=1024
RX_JUMBO=256

/sbin/ethtool -G eth0 rx $RX_SIZE rx-jumbo $RX_JUMBO
echo "ACTION==\"add|change\", SUBSYSTEM==\"net\", KERNEL==\"eth*|en*\", RUN+=\"/sbin/ethtool -G \$name rx ${RX_SIZE} rx-jumbo ${RX_JUMBO}\"" > /etc/udev/rules.d/61-net.ring.rules

This script runs during the pre-start lifecycle hook so this should allow the ring buffer sizes to persist VM recreates and reboots