"too many open files" error while application is polling messages in Tanzu Application Service for VMs

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

While an application is running and processing messages, it displays an error stating "too many open files." in Tanzu Application Service for VMs (TAS for VMs).

This error is happening at the Diego Cell level. We need a way to update the deployment to increase the limit size for the file descriptor.

A similar request for Cloud Foundry can be found here: met a problem, too many open files #163. The community suggests making the changes in the cloud config. However, the instance_file_descriptor_limit parameter is not found in the cloud config. We need to update the cloud config to make the changes permanent and persistent.

Note: This error is documented in "too many open files" in Tanzu Greenplum. However, this article pertains to the Greenplum Database (GPDB), not Tanzu Application Service for VMs. 

 

Environment

Product Version: 2.10

Resolution

The limit on file descriptors for applications running on Diego Cells is set at the container level and the limit is 16384.

If an application is hitting this limit, we recommend scaling the app horizontally by adding more application instances. This balances the workload across more containers and effectively gives you a higher open files limit. 

It is possible to adjust the open file limit on VMs in TAS for VMs by employing the os-conf bosh add-on. For more information, refer to Custom OS Configuration in BOSH. However, this would not increase the open file limit on containers, which is the limit seen by your application. The container limit is determined by the Cloud Controller (CC). The value comes from the default value defined in the CC spec file, but TAS for VMs doesn’t provide the UI to configure it. To see the spec file, refer to capi-release/spec - L949.

cc.instance_file_descriptor_limit:
  description: "The file descriptors made available to each app instance"
  default: 16384