Contour/Envoy in TAP not starting due to malformed IP address

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

In TAP v1.3 and above, the behaviour of Contour was changed and it's defaulting to IPv6 with IPv4 compatibility. This issue in particular can most commonly occur when using TAP on top of TKGi which uses IPv4 only. When upgrading to that version, it is possible that the Envoy pods can go down and become inaccessible. Since Envoy pods are down, Contour and TAP will fail to reconcile. The Envoy process fails to start correctly and as a result it is not listening at health check port 8002. The issue can present itself in the Envoy logs with the following error:

[2023-03-16 12:05:57.334][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 10 cluster(s), remove 2 cluster(s)
[2023-03-16 12:05:57.334][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 0 cluster(s), skipped 10 unmodified cluster(s)
[2023-03-16 12:15:09.584][1][warning][config] [source/common/config/grpc_subscription_impl.cc:126] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) ingress_http: malformed IP address: ::
ingress_https: malformed IP address: ::
stats-health: malformed IP address: ::

Environment

Product Version: 1.3

Resolution

When using the OOTB Contour Carvel package to install Contour, it uses IPv6 by default meaning that you'll need to be running TAP on a node that supports IPv6. That is the default setting for Contour, and only requires the IPv6 module to be loaded on the host, not for any type of IPv6 routes to be present. However, to work around this and deploy TAP in an environment where IPv6 is not available, one must create an overlay for Contour in order to override this setting and disable IPv6 so that it can run with IPv4. 

The overlay gets rid of IPv6-only addresses in the Contour configuration. We can still use IPv4, just the base image for nodes needs IPv6 kernel modules loaded etc. but otherwise we need an overlay to change the Contour/Envoy listeners to use addresses to IPv4. The following are the steps to create the overlay:

1) Create a secret with ipv4-overlay.yaml file which configures the listeners to use IPv4 addresses.

apiVersion: v1
kind: Secret
metadata:
  name: ipv4-overlay
  namespace: tap-install
stringData:
  ipv4-overlay.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"metadata":{"name":"contour"}, "kind": "Deployment"})
    ---
    spec:
      template:
        spec:
          containers:
            #@overlay/match by="name"
            - name: contour
              #@overlay/replace
              args:
              - serve
              - --incluster
              - '--xds-address=0.0.0.0'
              - --xds-port=8001
              - '--stats-address=0.0.0.0'
              - '--http-address=0.0.0.0'
              - '--envoy-service-http-address=0.0.0.0'
              - '--envoy-service-https-address=0.0.0.0'
              - '--health-address=0.0.0.0'
              - --contour-cafile=/certs/ca.crt
              - --contour-cert-file=/certs/tls.crt
              - --contour-key-file=/certs/tls.key
              - --config-path=/config/contour.yaml

2) Add the following lines to the tap-values.yaml file by overriding Contour start command with ipv4-overlay secret that we created in the step 1.

package_overlays:
- name: contour
  secrets:
  - name: ipv4-overlay

Once the overlay has been added to the tap-values.yaml file and applied with the below command, the components should reconcile and become operational.

tanzu package installed update tap -p tap.tanzu.vmware.com -v ${TAP_VERSION} --values-file tap-values.yaml -n tap-install