System slowdown or freeze during cluster service shutdown on Linux VMs running Guest Introspection agent (vmware-nsx-gi or vmware-nsx-gi-file)

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Cluster Databases such as Oracle RAC or applications that manage highly available (HA) clusters, work with multiple nodes. When these applications are required to be shut down for maintenance, they need to sync database and quiesce IO across nodes before the services can be shut down. Before IO can be fully quiesced there can be a short burst of file activity related to cluster sync operations. These operations are time constrained.  If the nodes on which this clustered application is installed is also running Guest Introspection agent then the file activity can be intercepted for inspection by file introspection component of NSX GI agent. Sudden burst of file activity, sometimes is seen slowing down the cluster application and in worst case cause a  system freeze.

Symptoms

• "kernel task hung" messages are seen on the terminal. dmesg shows "kernel task hung" message.
  From the affected Guest VM, the "dmesg" command is run from any terminal.
  
  
• System slows down when responding to user interactions.
  
  
• Eventually system freezes.

Environment

Guest Introspection agents (vmware-nsx-gi or vmware-nsx-gi-file) installed on a system where clustered application such as Oracle RAC or HA clusters are running

Cause

Depending upon the size (number of files in a burst), duration of the burst, continuous file IO can put resource pressure on Guest Introspection agent. If this leads to enough back pressure on system resources in Kernel's FANotify framework then such sustained file IO burst can cause a system freeze.

Resolution

This is a very specific problem observed mainly with clustered applications on shared file systems especially when cluster services are being shutdown. In lab tests this problem is seen to subside with time as the burst condition settles down. If this doesn’t happen, then in order to work around the stated problem, customers are advised to stop Guest Introspection agent before shutting down cluster services. Guest Introspection agent can be brought back up post stopping or restarting cluster services.

On Guest VM terminal, run following command to stop Guest Introspection agent service

systemctl stop vsepd

On Guest VM terminal, run following command to start Guest Introspection agent service

systemctl start vsepd

Additional Information

This is being tracked via Bug 3676015 and 3670504