Loss of network connectivity on Google Cloud Platform after applying security patch DSA-4208

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

PCF  installations are failing with an issue where NAT instances within GCP may upgrade the procps automatically and stop forwarding traffic. Depending on how NAT instances were created this may or may not affect a given foundation running on GCP. A foundation may also be running using public IPs and not using NAT, in which case the environment would not be vulnerable. 

How to tell if you’re affected:

• Any environment who is using an Ubuntu/Debian based NAT that runs as a VM may be affected by this.

• Any environment where NATs are built by terraform using this for the start-up script is affected:

  • #! /bin/bash
    

  • sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
    

  • sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    

• If you have set ip_forward enabled in /etc/sysctl.conf - the setting WILL persist across an upgrade of procps.

  • # Uncomment the next line to enable packet forwarding for IPv4

  • net.ipv4.ip_forward=1

• Environments using public IPs and not using NAT are not vulnerable

• Environments using managed NAT like AWS offers should be safe

Environment

Cause

Automatic security upgrade of the procps package for this CVE (https://www.debian.org/security/2018/dsa-4208) resets the state of /proc/sys/ipv4/ip_forwarding back to 0 which keeps the NAT instance from forwarding packets. In this state instances can talk to the NAT box and the NAT box can talk to the internet, but the NAT box will not forward packets. This effectively cuts off outbound access from the PCF deployment to the Internet. If this setting has been set in /etc/sysctl.conf it will be restored after an upgrade, but if it set via a startup script it will not be.

Resolution

When your environment is currently down please follow these steps:

• On the NAT box, check the value of /proc/sys/net/ipv4/ip_forward - it should be 1 and not 0.

  • cat /proc/sys/net/ipv4/ip_forward to check

  • sudo sysctl -w net.ipv4.ip_forward=1 to re-enable

• Check the last line of: cat /etc/sysctl.conf, if it is not "net.ipv4.ip_forward=1", then run the following command to ensure connectivity keep working after reboot:

  • echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf > /dev/null

• You can try rebooting your NAT instance to validate that connectivity is re-established

When your environment is vulnerable but not currently down, please follow these steps:

• Run `echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf > /dev/null`

• On the nat box, run `sudo apt-get update` and `sudo apt-get upgrade`. This may cause a network outage.

• Verify your procps package is greater than version 3.3.9-1ubuntu2.3