vMotion issues when Global Containers feature is enabled in NSX DFW
search cancel

vMotion issues when Global Containers feature is enabled in NSX DFW

book

Article ID: 314287

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:
This issue can cause two different sets of symptoms. In both cases, Global Containers (also known as Global Address Sets) is enabled.

Scenario 1 - vMotion failure:
  • vMotion fails and the following error is seen:
Failed waiting for data. Error 195887105. Failure.
The source detected that the destination failed to resume.
vMotion migration [-1062703306:5592608966896567683] failed: remote host <192.168.111.54> failed with status Failure.
vMotion migration [-1062703306:5592608966896567683] failed to asynchronously receive and apply state from the remote host: Failure.
Failed waiting for data. Error 195887105. Failure.
  • On vmkernel.log of the source host, the below errors are seen:
2020-06-22T15:15:49.724Z cpu4:4446496)Save state called for filter nic-4443130-eth0-vmware-sfw.2
2020-06-22T15:15:49.724Z cpu4:4446496)Exporting nic-4443130-eth0-vmware-sfw.2, Version 1000
2020-06-22T15:15:49.724Z cpu4:4446496)Create tbl failed: -1
2020-06-22T15:15:49.724Z cpu4:4446496)EXPORTTLVTABLES failed 28
2020-06-22T15:15:49.724Z cpu4:4446496)Unable to add tables. Total = 67452, used = 32, avail = 67420, need = 87
2020-06-22T15:15:49.724Z cpu4:4446496)Failed to retreive datapath save state data: Failure
Scenario 2 - Address sets lost after storage vMotion:
  • Storage vMotion succeeds. 
  • However, DFW rules are not being applied to the VM as expected, possibly causing connectivity to the VM to be lost.
  • During the vMotion, messages similar to the below are seen in vmkernel.log of the ESXi host.
2020-06-22T12:10:40.109Z cpu7:4439212)Save state called for filter nic-4438977-eth0-vmware-sfw.2
2020-06-22T12:10:40.109Z cpu7:4439212)Exporting nic-4438977-eth0-vmware-sfw.2, Version 1000
2020-06-22T12:10:40.109Z cpu7:4439212)Create tbl failed: -1
  • Attempting to retrieve the address sets on the VM's filter will give the below error.
# vsipioctl getaddrsets -f nic-4443130-eth0-vmware-sfw.2
ERROR: failed to get size of addrset ip-vm-186: ioctl failed
#



Cause

The Global Containers feature (also known as Global Address Sets) is only available starting 6.4.0. It makes the ESXi host keep only one copy of the DFW address sets per host instead of per filter.

When Global Containers feature is enabled, global address sets fail to be correctly exported during a vMotion. This can cause the vMotion to fail, or to succeed but the VM to lose its address sets.

If global and local tables exist, the vMotion blob creation fails on the source host. This stops the vMotion from happening. If only global tables exist, then the vMotion blob is created but with no global tables present in the blob.

Resolution

This issue affects all pre-6.4.7 versions and is fixed in 6.4.7 and later.

Workaround:
To restore access in case the address sets are lost, perform any change on DFW and re-publish. This will cause the address sets to be pushed back to the hosts.

If vMotions are failing, disable Global Containers to allow vMotions to happen.

To disable Global Containers:
1. Retrieve the current setting with: 
GET /api/4.0/firewall/config/globalconfiguration
Example of the output: 
<?xml version="1.0" encoding="UTF-8"?>
<globalConfiguration>
    <layer3RuleOptimize>false</layer3RuleOptimize>
    <layer2RuleOptimize>true</layer2RuleOptimize>
    <tcpStrictOption>false</tcpStrictOption>
    <enableGlobalContainers>true</enableGlobalContainers>
    <ruleStatsDisabled>false</ruleStatsDisabled>
</globalConfiguration>
2. Change the value of enableGlobalContainers to false in the retrieved result and run: 
PUT /api/4.0/firewall/config/globalconfiguration
Body: 
<globalConfiguration>
    <layer3RuleOptimize>false</layer3RuleOptimize>
    <layer2RuleOptimize>true</layer2RuleOptimize>
    <tcpStrictOption>false</tcpStrictOption>
    <enableGlobalContainers>false</enableGlobalContainers>
    <ruleStatsDisabled>false</ruleStatsDisabled>
</globalConfiguration>






Additional Information

Example of a filter with a local address set. In this example there is a local address set because a context aware firewall rule is used. 
# vsipioctl getaddrsets -f nic-4443130-eth0-vmware-sfw.2
addrset is shared for this filter
global addrset
No address sets.

local addrset
addrset attr_1007_1_APP_ID {
# generation number: 1592838483960
# realization time : 2020-06-22T15:08:05
int attr 81 APP_HTTP,
}
Example of a filter with no local address set. 
# vsipioctl getaddrsets -f nic-4443130-eth0-vmware-sfw.2
addrset is shared for this filter
global addrset
addrset ip-vm-186 {
# generation number: 1592828630729
# realization time : 2020-06-22T12:32:09
ip 172.16.50.11,
ip fe80::250:56ff:feab:f8c3,
}


local addrset
No address sets.


Powered by Wolken Software