NSX-T Edge Flow Cache

Products

• VMware NSX

Issue/Introduction

• Flow Cache is a forwarding performance accelerator.
• The fundamental rationale of Flow Cache is that all packets belonging to the same flow shall share the same processing result, and hence the processing outcomes applied to the first packet can be applied to later packets in the same flow, as long as those packets can be properly identified by a flow cache key.
• A fast path allows traffic to be forwarded based on Flow Cache information instead of the slow path alternative of CPU lookup.
• By default, Flow Cache is enabled on the NSX-T Edge to provide maximum performance.
• If investigating a data path issue, it may be desirable to understand if the problem is also present when Flow Cache is disabled, this article provides information on how to disable and re-enable Flow Cache.

Environment

VMware NSX-T Data Center
VMware NSX 

Resolution

Flow Cache can be disabled from the admin cli as follows
>set dataplane flow-cache disabled

However this change does not take effect until a dataplane restart 
>restart service dataplane

Similarly it can be re-enabled from admin cli, once more followed by a dataplane restart
>set dataplane flow-cache enabled

If the purpose of disabling Flow Cache is to test if it may be contributing to some data path issue, then this is not a good isolated test since restarting the dataplane in itself may resolve the issue through a Gateway failover or some other reason.

From NSX 4.0.0.1 and above, it is possible to disable and enable Flow Cache dynamically from root shell of the Edge without any requirement to restart the dataplane service.

Disable Flow Cache (root shell)
#edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/disable

Enable Flow Cache (root shell)
#edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/restore

This command enables Flow Cache by restoring the original Flow Cache configuration present when dataplane was last started.

Note it will also reset the Flow Cache statistics as seen using the admin cli
>get dataplane flow-cache stats 0

Confirm Flow Cache status  (admin shell)

> get dataplane flow-cache config 
Enabled            : true
Mega_hard_timeout_ms: 4955
Mega_size          : 256
Mega_soft_timeout_ms: 2994
Micro_size         : 256

Note: The command's edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/disable and edge-appctl -t /var/run/vmware/edge/dpd.ctl flow_cache/restore are runtime command's and will not persist after a reboot or dataplane service restart.

The admin/nsxcli command set dataplane flow-cache <enabled|disabled> should be used to make the configuration persist.

To view the current configured state use the admin/nsxcli command get dataplane flow-cache config

Get  Flow Cache stats  (admin shell)

> get dataplane flow-cache stats
Mega
Core              : 0
Total             : 0/262144
Dont cache        : 106791
Hit rate          : n/a
Hits              : 0
Insertions        : 0
Misses            : 0
Skipped           : 61071
Bucket collisions : 0
Key collisions    : 0

Core              : 1
Total             : 0/262144
Dont cache        : 5
Hit rate          : n/a
Hits              : 0
Insertions        : 0
Misses            : 0
Skipped           : 10489
Bucket collisions : 0
Key collisions    : 0

Micro
Core              : 0
Total             : 0/262144
Dont cache        : 0
Hit rate          : n/a
Hits              : 0
Insertions        : 0
Misses            : 0
Skipped           : 61071
Bucket collisions : 0
Key collisions    : 0

Core              : 1
Total             : 0/262144
Dont cache        : 0
Hit rate          : n/a
Hits              : 0
Insertions        : 0
Misses            : 0
Skipped           : 10489
Bucket collisions : 0
Key collisions    : 0

These stats are not intended to be used for day to day performance monitoring and are intended for advanced troubleshooting when dataplane issues occur. In general the higher the hit rate the better flow-cache is performing. Key Collisions are normal and expected due to the hashing method of flow cache and are not a sign of an issue.