"cf push" of .Net apps is very slow due to unreliable syslog service

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:
In some cases, cf push of .NET apps is extremely slow (can take up to  ~20 minutes to push an app). “cf push” operations can be very slow on Windows cells, including downloading buildpacks, staging the app and starting the app.

Environment

Cause

In one particular instance, it was discovered that if the syslog server endpoint is unreliable and if the network transport is over tcp, it can cause Diego components on the Windows cell to respond slowly. This will, in turn affect staging, starting and downloading Buildpack operations.

Resolution

Change the syslog settings in ERT/PAS UI -> System Logging -> Transport Protocol  to UDP.

The Windows Runtime components use an executable called `pipe.exe` to log to stdout and to a syslog endpoint (if a syslog endpoint is configured in the ERT/PAS Tile, this setting gets inherited to the PAS for Windows (PASW) components. If the syslog endpoint is unreliable, then the write operation will block the calling process as well (in our case, the rep process) This has a cascading effect on downloading Buildpacks, staging and starting operations.

When the syslog transport is set to UDP, the write operation is no longer blocked due to inherent connectionless nature of the UDP transport protocol.