Flow fails to display in the UI when system memory gets overloaded

Products

• VMware NSX

Issue/Introduction

Symptoms:

• Displaying flow in the NSX Intelligence User Interface (UI) fails.
• In NSX Intelligence node console, you see an error message similar to:
  
  INFO: task kworker/5:1:5539 blocked for more than 120 seconds.
  Not tainted 2.6.32-431.11.2.el6.x86_64 #1
  "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

This issue occurs as after memory for file system caching has been used up, the file system flushes all outstanding data to disk causing all following IOs going synchronous.

For flushing out this data to disk, this there is a time limit of 120 seconds by default. In this case, the IO subsystem is not fast enough to flush the data within 120 seconds. As IO subsystem responds slowly and more requests are served, the System Memory gets filled up resulting in the above error.

Resolution

To resolve this issue:

Wait for system to stabilize. It might take 20-30 minutes.

Workaround:
To work around this issue, leave the system idle and wait for it to recover.