Windows Diego Cell consumes 100% swap usage and goes to a failing state

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Windows Diego cell consumes 100% swap usage. You can check this on Ops manager UI -> TASW Tile -> Status Tab or by using bosh vms --vitals. 

The Windows Diego cell would go from failing state intermittently. 

One other symptom prom_scraper job would throw sample error below:
 

pas-windows-743bd9fca357c14d590a.windows_diego_cell.ee9652d5-571a-42b1-ace6-bc97cd358d47-20220520-060832-370228437/prom_scraper_windows/prom_scraper/job-service-wrapper.err.log:fatal error: runtime: cannot allocate memory

Environment

Product Version: 2.12

Resolution

As of time of writing version up to TASW v2.13.5(current version) are affected.  The high RAM/swap usage suggests that the apps are leaking memory. Windows containers do not kill applications when they exceed their memory limit – instead, Windows pages the memory to disk. We believe that this is the reason the swap memory is filled up.

Here are workarounds:

• Recreating Windows Diego cell - This is not the best experience but this will put a stop on the symptoms temporarily
• Set a Diego Cell memory capacity of 3GB less than the current Windows Diego VM Memory Capacity. You can configure this by going to TASW Tile -> Settings -> Advanced Features -> Diego memory Capacity Field then apply changes for new configuration to take effect. The default value is "auto" which advertises full system memory, and doesn’t reserve anything for system resources. Putting a limit, this will make sure there is some memory reserved for system resources