Workaround for CVE-2019-11477, CVE-2019-11478 on NSX for vSphere 6.4.5 and prior versions

Products

• VMware NSX

Issue/Introduction

CVE-2019-11477, CVE-2019-11478 have been determined to affect NSX for vSphere 6.4.5 and prior versions. These vulnerabilities, their affect on VMware products, and VMware’s overall response is documented in VMSA-2019-0010. Please review this advisory before continuing as there may be considerations outside the scope of this particular document including permanent solutions.

The NSX for vSphere team has determined that the aforementioned issues can be mitigated by performing the steps detailed in the resolution section of this article. This workaround is meant to be a temporary solution only - permanent fixes will be released as soon as they are available.

Resolution

All NSX-V components has the fix for these CVE-2019-11477, CVE-2019-11478 in the upcoming NSX-V 6.4.6 and later versions.  Customers can upgrade to 6.4.6 & later versions for permanent resolution for the above mentioned CVE's.

To implement the workaround for CVE-2019-11477, CVE-2019-11478 perform the following steps on NSX-V Manager, Controllers, Edge and USVM Virtual Machines.
 

Workaround Procedure to be executed on  NSX-V Manager VM

1. Login to the NSX Manager node as root via SSH or Console. Please use VMware KB2149630 for NSX manager root access.
2. Run the following commands to Check the TCP SACK status whether its enabled or disabled.

Note: Its enabled by Default.
sysctl -a | grep tcp_sack
net.ipv4.tcp_sack = 1

3. Add the below config by editing /etc/sysctl.conf using vi editor and save it.

net.ipv4.tcp_sack = 0

4. Reload the sysctl.conf using sysctl -p command.
5. Verify changes are updated in the sysctl.conf file and TCP SACK is disabled successfully:

less sysctl.conf | grep sack
Example: net.ipv4.tcp_sack = 0

sysctl -a | grep tcp_sack
Example: net.ipv4.tcp_sack = 0

 

Workaround Procedure to be executed on NSX-V Controller VM's

1. Login in to root mode of the controller .To switch to the root user on any controller node, we first need the root password for the specific controller. Please follow below steps to get root access to controller 

Root Login steps for NSX-V Controller Nodes:

• Login in to root mode of NSX Manager using KB2149630.
• Look for the controller id in the Networking & Security Tab in the vSphere (Web) client under the controller deployment section (Networking & Security > Installation & Upgrade > Management > NSX Controller Nodes).
• Execute the following command in the Linux shell of the NSX Manager:

/home/secureall/secureall/sem/WEB-INF/classes/GetNvpApiPassword.sh controller-NN
 

Note: Replace controller-NN with the correct controller id. For example: controller-12)

In the last line you will find the root password for this controller node:

Now, login as “admin” via SSH on the controller

Type the following command:

: debug os-shell

• Enter the root password which was displayed on the NSX Manager shell.Now you are in root mode of controller

2. Execute below IP table rules on all 3 controllers to implement the workaround for CVE-2019-11477 and CVE-2019-11478:

iptables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP
ip6tables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP

3. Verify IP table rules are added successfully using below command:

iptables -L | grep tcpmss

Example: DROP       tcp  --  anywhere             anywhere             tcp flags:SYN/SYN tcpmss match 1:500

4. Save the IP Table rules for restore in case of controller reboot:

iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6

Note: After Reboot of controller, above iptables rules added are not getting persisted by default. If controllers are rebooted by any chance. After Reboot completion, customer need to execute below commands in root mode: 
iptables-restore < /etc/iptables/rules.v4
iptables-restore < /etc/iptables/rules.v6

 

Workaround  Procedure to be executed on NSX-V Edge VM's

1. Login to Edge in root mode. To switch to the root user on Edge VM we first need the root password . Please follow below steps to get root access to Edge VM.

Root Login steps for Edge VM:

• First, we need the generated root password from the NSX Manager and for this we need the ESG. This ID can be found in the vSphere (Web) Client under Networking & Security > NSX Edges in the first column of the displayed list under ID.
• Once we know the ID, we can execute the following command in the Linux shell of the NSX Manager:

'/home/secureall/secureall/sem/WEB-INF/classes/GetSpockEdgePassword.sh edge-NN

Note: Replace edge-NN to the correct edge ID.

• Login with the admin user in the local console (web console) for activating the engineering mode. It can’t be enabled in a remote session.
• Switch to enable mode with the command “en” and enter the admin passwordAnd activate the engineering mode with the following command:

debug engineering mode enable

• After the engineer mode was enabled in a local console, you can login via SSH with the admin user Switch to enable mode with the command en and enter the admin password.
• Finally you can get root with the following command:

st en

• Enter the root password for the Edge Gateway or Distributed Logical Router.

2. Run the following commands to Check the TCP SACK  status whether its enabled or disabled:

Note: Its enabled by Default.

sysctl -a | grep tcp_sack

Example: net.ipv4.tcp_sack = 1

3. Edit the opt/vmware/vshield/templates/sysctl_size.conf file with a text editor:

Note: There is a separate file for each formfactor: sysctl_compact.conf, sysctl_large.conf; sysctl_xlarge.conf; sysctl_quadlarge.conf. Select the file as per the deployed edge formfactor.

4. Change the net.ipv4.tcp_sack = 1 to net.ipv4.tcp_sack = 0, then save and close the file.
5. Edit the /etc/sysctl.conf file with a text editor:

Note: This config file gets regenerated at every boot from above file.

6. Change the net.ipv4.tcp_sack = 1 to net.ipv4.tcp_sack = 0, then save and close the file.
7. Reload the updated sysctl config using below command:

sysctl -p  /etc/sysctl.conf

8. Verify the changes applied using the below command:

sysctl -a | grep tcp_sack -> o/p : net.ipv4.tcp_sack = 0

Note: These changes persist across reboots and Power-Offs however NOT in case of Edge appliance gets changed as part of Redeploy/FormFactorChange.
 

 

Workaround  Procedure to be executed on USVM's ( GI SVM) for customers using Guest Introspection Feature

1. Log in to root mode of SVM. To switch to the root user on USVM we first need the root password . Please follow below steps to get root access to USVM

Root Login steps for GI USVM:

• Enter the Linux console of the NSX Manager as root user using KB2149630.
• Run the following command to obtain the root password of the GI-SVM(s):

/home/secureall/secureall/sem/WEB-INF/classes/GetEpsecAppliancePassword.sh

• After that you can log on as root user in a local console.

2. Execute below IP table rules on the SVM:

iptables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP
ip6tables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP

3. Verify IP table rules are added successfully using below command:

iptables -L | grep tcpmss

Example: DROP       tcp  --  anywhere             anywhere             tcp flags:SYN/SYN tcpmss match 1:500

4. Save the IP Table rules for restore in case of svm reboot:

iptables-save > /etc/iptables/rules.v4
iptables-save > /etc/iptables/rules.v6

5. After Reboot of SVM, above iptables rules added are not getting persisted by default. If SVM  was rebooted by any chance. After Reboot completion, customer need to execute below commands in root mode to restore the iptables:

iptables-restore < /etc/iptables/rules.v4
iptables-restore < /etc/iptables/rules.v6

 

Remove the Workarounds

To remove the workarounds for CVE-2019-11477, CVE-2019-11478 at a later time perform the following steps:

1. Enable TCP SACK on NSX Manager And Edge in the respective sysctl*.conf files where it was enabled before:

• Add the below config by editing /etc/sysctl.conf using vi editor and save it:

net.ipv4.tcp_sack = 1

• Reload the sysctl.conf using sysctl -p command.
• Verify changes are updated in the sysctl.conf file and TCP SACK is enabled:

less sysctl.conf | grep sack

Example: net.ipv4.tcp_sack = 1

2. Delete the IP table Rules added for the workaround by executing below commands on NSX Controller and USVM:

iptables -D INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP
ip6tables -D INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP

Additional Information

For up-to-date information on CVE-2019-11477, CVE-2019-11478 as well as future security information please add your email address to the "Sign up for Security Advisories" window found in VMSA-2019-0010.

Impact/Risks:

Warning

This workaround is applicable ONLY to NSX for vSphere 6.4.5 and prior versions Do not apply this workaround to other VMware products.
 

Functionality Impacts

There will be no functionality impact on NSX  that will result from applying the workaround.