NATS VM failing with "No servers available for connection" in TAS 4.x+

Products

• VMware Tanzu Application Service

Issue/Introduction

When using TAS 4.x+, you may see the following error repeated in the NATS VMs in the CF Deployment:

failed to connect to NATS server: nats: no servers available for connection

This causes the NATS vm to fail, which can cause other issues on the platform.

Environment

TAS 4.x+

Cause

This is an issue on the VM OS level. If you check these files on the NATS VM:

cat /proc/sys/net/netfilter/nf_conntrack_count

cat /proc/sys/net/netfilter/nf_conntrack_max

You should see that the value for /proc/sys/net/netfilter/nf_conntrack_count is greater or equal to than /proc/sys/net/netfilter/nf_conntrack_max. This prevents new connections from being established. 

Resolution

The value for /proc/sys/net/netfilter/nf_conntrack_max is based on the RAM available to the VM. This means to resolve the issue, you'll need to scale the NATS VM type to have more memory.