NSX-T Edge Flow Cache

Products

• VMware NSX

Issue/Introduction

• Flow Cache is a forwarding performance accelerator.
• The fundamental rationale of Flow Cache is that all packets belonging to the same flow shall share the same processing result, and hence the processing outcomes applied to the first packet can be applied to later packets in the same flow, as long as those packets can be properly identified by a flow cache key.
• A fast path allows traffic to be forwarded based on Flow Cache information instead of the slow path alternative of CPU lookup.
• By default, Flow Cache is enabled on the NSX-T Edge to provide maximum performance.
• If investigating a data path issue, it may be desirable to understand if the problem is also present when Flow Cache is disabled, this article provides information on how to disable and re-enable Flow Cache.

Environment

VMware NSX-T Data Center

Resolution

Flow Cache can be disabled from the admin cli as follows
>set dataplane flow-cache disabled

However this change does not take effect until a dataplane restart 
>restart service dataplane

Similarly it can be re-enabled from admin cli, once more followed by a dataplane restart
>set dataplane flow-cache enabled

If the purpose of disabling Flow Cache is to test if it may be contributing to some data path issue, then this is not a good isolated test since restarting the dataplane in itself may resolve the issue through a Gateway failover or some other reason.

From NSX 4.0.0.1 and above, it is possible to disable and enable Flow Cache dynamically from root shell of the Edge without any requirement to restart the dataplane service.

Disable Flow Cache (root shell)
#edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/disable

Enable Flow Cache (root shell)
#edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/restore

This command enables Flow Cache by restoring the original Flow Cache configuration present when dataplane was last started.

Note it will also reset the Flow Cache statistics as seen using the admin cli
>get dataplane flow-cache stats 0

Confirm Flow Cache status  (admin shell)

> get dataplane flow-cache config 
Enabled            : true
Mega_hard_timeout_ms: 4955
Mega_size          : 256
Mega_soft_timeout_ms: 2994
Micro_size         : 256

Note: The command's edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/disable and edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/restore are runtime command's and will not persist after a reboot or dataplane service restart.

The admin/nsxcli command set dataplane flow-cache <enabled|disabled> should be used to make the configuration persist.

To view the current configured state use the admin/nsxcli command get dataplane flow-cache config