Aria Automation creates DNS Flood with "contour" requests.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• After upgrading Aria Automation to version 8.18.1 the DNS servers are receiving thousands of IPv6 requests.
• The file /var/log/vmware/prelude/upgrade-report-latest may contain entries stating:
  • "Upgrade failed and left the system in a non-working state. Check the error report below to correct the problem."
  • "Pod: contour-envoy-* is not in Ready or Completed state. All pods must be in either of these states."

Environment

• VMware Aria Automation 8.18.1

Cause

• The new ingress controller "contour" was introduced in Aria Automation 8.18.1 in place of the old "traefik".

Resolution

This procedure applies to both single-node and clustered environments. In a clustered environment, it should be performed on each node.

  SSH into the Aria Automation appliance.
    
    Edit the /opt/charts/contour/values.yaml file by applying "dns-lookup-family: v4" under cluster section. Add/modify the configInline section as follows (Note: Add/modify only the "dns-lookup-family: v4"):
         configInline:
           cluster:
              dns-lookup-family: v4
              ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
      

          Edit the /opt/charts/contour/templates/envoy/daemonset.yaml file by adding/modifying bootstrap parameter: "- --dns-lookup-family=v4" (Note: only append the "---dns-lookup-family=v4", the other fields remain unchanged). Here is how to append to the initContainers/args section:

      initContainers/args:
        - command:
            - contour
          args:
            ...
            - --dns-lookup-family=v4
            ...

Restart Aria Automation services with the following command:
  /opt/scripts/deploy.sh on any of the nodes.
 
  Post-validation:

    SSH into any node of the Aria Automation appliance.
  Run the following commands:

    1. kubectl -n prelude get daemonset contour-envoy -o json | jq -r '.spec.template.spec.initContainers[0].args'
  The output should be similar to:
         ----
         [
         "bootstrap",
         "/config/envoy.json",
         "--xds-address=contour",
         "--xds-port=8001",
         "--resources-dir=/config/resources",
         "--envoy-cafile=/certs/ca.crt",
         "--envoy-cert-file=/certs/tls.crt",
         "--envoy-key-file=/certs/tls.key",
         "--dns-lookup-family=v4"
         ]
         ----

    2. kubectl -n prelude get cm contour -o json | jq -r '.data."contour.yaml"'
  The output should be similar to:
         ----
           cluster:
         dns-lookup-family: v4
         disablePermitInsecure: false
         tls:
         fallback-certificate:
            name: contour-ingress-cert-and-key
            namespace: prelude
         ----

 

Additional Information

Once deploy.sh has been executed on the node where the changes are done, it's already applied in the cluster and is effective. However, the chart yaml files (/opt/charts/contour/values.yaml and /opt/charts/contour/templates/envoy/daemonset.yaml) need to be updated on the other nodes in the cluster as well, in case in future deploy.sh is run on one of them.